Freehand ultrasound imaging systems and methods for guiding fine elongate instruments

ABSTRACT

An ultrasound system has an ultrasound transducer equipped with a position marker and a needle equipped with a position marker. The position markers allow the position and orientation of the transducer and needle to be determined. Displays indicating a projection of the longitudinal axis of the needle onto a plane of an ultrasound image acquired via the transducer and displays indicating a projection of a reference position onto a plane of an ultrasound image acquired via the transducer are provided. Displays indicating in real-time whether the needle is substantially within the plane of the ultrasound image are provided.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of application No. 61/180,050 filedon 20 May 2009 and entitled ULTRASOUND SYSTEMS INCORPORATING SPATIALPOSITION SENSORS AND ASSOCIATED METHODS, which is hereby incorporatedherein by reference.

TECHNICAL FIELD

This invention relates to ultrasound imaging. The invention hasparticular application in the field of medical ultrasonography andoncology.

BACKGROUND

Ultrasound imaging is widely used in a range of medical applications.One area in which ultrasound imaging is used is to guide biopsyprocedures. A biopsy typically involves identifying an abnormality ofinterest, such as suspicious solid mass, a distortion in the structureof a body tissue, or an area of abnormal tissue change. A needle orother fine member may be inserted into the abnormality and used towithdraw a small tissue sample for investigation.

Various types of needles may be used for biopsies. In fine needleaspiration, small hollow needles are used to extract cells from anabnormality. A core needle is a larger diameter needle which may be usedto withdraw larger samples of tissue. Vacuum assisted devices may beused to collect multiple tissue samples during one needle insertion. Insome cases ultrasound is used to assist in placing a guide wire into anabnormality to assist a surgeon in locating the abnormality for asurgical biopsy.

A problem with the use of ultrasound to guide a needle or wire in any ofthese procedures, or like procedures, is that the thin needles are oftenvery difficult to see in an ultrasound image. This makes it difficultfor a person taking the biopsy to ensure that the needle has reached itstarget. Also, guiding the needle to place the tip of the needle at anarea of abnormality shown in an ultrasound image takes a significantamount of skill because the image does not always provide good feedbackto the practitioner regarding exactly where the needle is placed and howthe needle should be manipulated to cleanly enter the abnormality. Also,the needle may not be visible in the ultrasound image because the needleis out of the plane of the ultrasound image.

The following US patents and publications disclose technology in thegeneral field of this invention:

-   U.S. Pat. No. 7,221,972 to Jackson et al.;-   U.S. Pat. No. 5,161,536 to Vilkomerson et al.;-   U.S. Pat. No. 6,216,029 to Palteili;-   U.S. Pat. No. 6,246,898 to Vesely et al.;-   U.S. Pat. No. 6,733,458 to Stein et al.;-   U.S. Pat. No. 6,764,449 to Lee et al.;-   U.S. Pat. No. 6,920,347 to Simon et al.;-   2004/0267121 to Sarvazyan et al.;-   WO 94/24933 to Bucholz;-   WO 97/03609 to Paltieli;-   WO 99/27837 to Paltieli et al.;-   WO 99/33406 to Hunter et al.;-   Freehand 3D Ultrasound Calibration: A Review, P-W. Hsu, R. W.    Prager A. H. Gee and G. M. Treece CUED/F-INFENG/TR 584, University    of Cambridge Department of Engineering, December 2007

SUMMARY

The following aspects and embodiments thereof are described andillustrated in conjunction with systems, apparatus and methods which aremeant to be exemplary and illustrative, not limiting in scope. Invarious embodiments, one or more of the above-described problems havebeen reduced or eliminated, while other embodiments are directed toother improvements.

An aspect of the invention provides an ultrasound system for use indirecting a fine elongate instrument, the instrument having alongitudinal axis and a tip, towards a reference position located in abody, the ultrasound system comprising an ultrasound transducer operableto receive ultrasound echo signals returning from a portion of the body,a position sensing system operable to monitor a spatial location andorientation of the instrument and a spatial location and orientation ofthe ultrasound transducer, a controller communicatively coupled to theultrasound transducer and the position sensing system, and a displaycommunicatively coupled to the controller, wherein the controller isconfigured to generate a two dimensional ultrasound image based on theultrasound echo signals display the ultrasound image on the display,determine a location of the portion of the body depicted in theultrasound image based on the spatial location and orientation of theultrasound transducer, and generate on the display a markercorresponding to a projection of the reference position onto the planeof the ultrasound image.

In some embodiments according to this aspect, the projection of thereference position onto the plane of the ultrasound image comprises anorthogonal projection onto the plane of the ultrasound image.

In some embodiments according to this aspect, the projection of thereference position onto the ultrasound image comprises a projectionparallel to the longitudinal axis of the instrument.

In some embodiments according to this aspect, the controller isconfigured to compute a distance between the reference position and theplane of the ultrasound image, and to indicate on the display thedistance between the reference position and the plane of the ultrasoundimage by applying a coded color and/or a coded luminosity to the markercorresponding to the projection of the reference position onto the planeof the ultrasound image. In some embodiments according to this aspect,the controller is configured to compute a distance between the referenceposition and the plane of the ultrasound image, and to indicate on thedisplay the distance between the reference position and the plane of theultrasound image using a coded marker forming part of the marker of theprojection of the reference position onto the plane of the ultrasoundimage.

In some such embodiments, the distance between the reference positionand the plane of the ultrasound image comprises a distance along a linethat is orthogonal to the plane of the ultrasound image. In other suchembodiments, the distance between the reference position and the planeof the ultrasound image comprises a distance along a line that isparallel to the longitudinal axis of the instrument.

In some embodiments according to this aspect, the controller isconfigured to determine a location of a reference plane that containsthe reference position and is parallel to the plane of the ultrasoundimage, determine a location of an axis-reference position planeintersection of the longitudinal axis of the instrument with referenceplane, and to generate on the display a marker corresponding to aprojection of the axis-reference position plane intersection onto theplane of the ultrasound image. In some such embodiments, the projectionof the axis-reference position plane intersection onto the plane of theultrasound image comprises an orthogonal projection onto the plane ofthe ultrasound image.

In some embodiments according to this aspect, the controller isconfigured to determine a location of an axis-image plane intersectionof the longitudinal axis of the instrument with the plane of theultrasound image, and to generate on the display a marker indicating thelocation of the axis-image plane intersection. In some such embodiments,the controller is configured to determine an angle between thelongitudinal axis of the instrument and the plane of the ultrasoundimage, and to indicate on the display the angle between the longitudinalaxis of the instrument and the plane of the ultrasound image by applyinga coded color and/or a coded luminosity to the marker indicating thelocation of the axis-image plane intersection.

In some embodiments according to this aspect, the controller isconfigured to determine an angle between the longitudinal axis of theinstrument and the plane of the ultrasound image, and to indicate on thedisplay the angle between the longitudinal axis of the instrument andthe plane of the ultrasound image using a coded marker comprised in themarker indicating of the location of the axis-image plane intersection.In some such embodiments, the coded marker comprises two lines meetingat a vertex and forming an angle corresponding to the angle between thelongitudinal axis of the instrument and the plane of the ultrasoundimage, the angle of the marker bisected by an orthogonal projection ofthe longitudinal axis of the instrument onto the plane of the ultrasoundimage, the vertex located at the axis-image plane intersection.

In some embodiments according to this aspect, the controller isconfigured to determine a location of the tip of the instrument based onthe spatial location and orientation of the instrument, compute adistance between the location of the tip of the instrument and the planeof the ultrasound image, and to indicate on the display the distancebetween the tip of the instrument and the plane of the ultrasound imageby applying a coded color and/or a coded luminosity to the markerindicating indication of the location of the axis-image planeintersection.

In some embodiments according to this aspect, the controller isconfigured to determine a location of the tip of the instrument based onthe spatial location and orientation of the instrument, compute adistance between the location of the tip of the instrument and the planeof the ultrasound image, and to indicate on the display the distancebetween the tip of the instrument and the plane of the ultrasound imageby a coded size of the marker indicating the location of the axis-imageplane intersection.

In some such embodiments, the distance between the tip of the instrumentand the ultrasound image comprises the distance along a line from thetip of the instrument to the plane of the ultrasound image that isorthogonal to the plane of the ultrasound image. In other suchembodiments, the distance between the tip of the instrument and theplane of the ultrasound image comprises the distance along a line fromthe tip of the instrument to the plane of the ultrasound image that isparallel to the longitudinal axis of the instrument.

Another aspect of the invention provides a method for generating adisplay useful in directing a fine elongate instrument, the instrumenthaving a longitudinal axis and a tip, towards a reference positionlocated in a body, the method comprising receiving ultrasound echosignals returning from a portion of the body, monitoring a spatiallocation and orientation of the instrument and a spatial location andorientation of the ultrasound transducer, generating a two dimensionalultrasound image based on the ultrasound echo signals, displaying theultrasound image on the display, determining a location of the portionof the body depicted in the ultrasound image based on the spatiallocation and orientation of the ultrasound transducer, and generating onthe display a marker corresponding to a projection of the referenceposition onto the plane of the ultrasound image.

In some embodiments according to this aspect, the projection of thereference position onto the plane of the ultrasound image comprises anorthogonal projection onto the plane of the ultrasound image. In otherembodiments according to this aspect, the projection of the referenceposition onto the ultrasound image comprises a projection parallel tothe longitudinal axis of the instrument.

In some embodiments according to this aspect, the method comprisescomputing a distance between the reference position and the plane of theultrasound image, and indicating on the display the distance between thereference position and the plane of the ultrasound image by applying acoded color and/or a coded luminosity to the indication of theprojection of the reference position onto the plane of the ultrasoundimage.

In some embodiments according to this aspect, the method comprisescomputing a distance between the reference position and the plane of theultrasound image, and indicating on the display the distance between thereference position and the plane of the ultrasound image using a codedmarker forming part of the indication of the projection of the referenceposition onto the plane of the ultrasound image.

In some such embodiments, the distance between the reference positionand the plane of the ultrasound image comprises a distance along a linethat is orthogonal to the plane of the ultrasound image. In other suchembodiments, the distance between the reference position and the planeof the ultrasound image comprises a distance along a line that isparallel to the longitudinal axis of the instrument.

In some embodiments according to this aspect, the method comprisesdetermining a location of an axis-reference position plane intersectionof the longitudinal axis of the instrument with a plane that containsthe reference position and is parallel to the plane of the ultrasoundimage, and generating on the display a marker corresponding to aprojection of the axis-reference position plane intersection onto theplane of the ultrasound image. In some such embodiments, the projectionof the axis-reference position plane intersection onto the plane of theultrasound image comprises an orthogonal projection onto the plane ofthe ultrasound image.

In some embodiments according to this aspect, the method comprisesdetermining a location of an axis-image plane intersection of thelongitudinal axis of the instrument with the plane of the ultrasoundimage, and generating on the ultrasound image a marker indicating thelocation of the axis-image plane intersection. In some such embodiments,the method comprises determining an angle between the longitudinal axisof the instrument and the plane of the ultrasound image, and indicatingon the display the angle between the longitudinal axis of the instrumentand the plane of the ultrasound image by applying a coded color and/orcoded luminosity to the indication of the location of the axis-imageplane intersection. In some embodiments, the method comprises indicatingon the display the angle between the longitudinal axis of the instrumentand the plane of the ultrasound image using a coded marker forming partof the indication of the location of the axis-image plane intersection.In some such embodiments, the coded marker comprises two lines meetingat a vertex and forming an angle corresponding to the angle between thelongitudinal axis of the instrument and the plane of the ultrasoundimage, the angle of the marker bisected by an orthogonal projection ofthe longitudinal axis of the instrument onto the plane of the ultrasoundimage, the vertex located at the axis-image plane intersection.

In some embodiments according to this aspect, the method comprisesdetermining a location of the tip of the instrument based on the spatiallocation and orientation of the instrument, computing a distance betweenthe location of the tip of the instrument and the plane of theultrasound image, and indicating on the display the distance between thetip of the instrument and the plane of the ultrasound image using acoded color and/or coded luminosity applied to the indication of thelocation of the axis-image plane intersection. In some embodiments, themethod comprises indicating on the display the distance between the tipof the instrument and the plane of the ultrasound image using a codedsize of the marker indicating the location of the axis-image planeintersection.

In some embodiments, the distance between the tip of the instrument andthe ultrasound image comprises a distance along a line from the tip ofthe instrument to the plane of the ultrasound image that is orthogonalto the plane of the ultrasound image. In other embodiments, the distancebetween the tip of the instrument and the plane of the ultrasound imagecomprises a distance along a line from the tip of the instrument to theplane of the ultrasound image that is parallel to the longitudinal axisof the instrument.

Yet another aspect of the invention provides an ultrasound system foruse in directing a fine elongate instrument in a body, the instrumenthaving a longitudinal axis and a tip, the ultrasound system comprisingan ultrasound transducer operable to receive ultrasound echo signalsreturning from a portion of the body, a position sensing system operableto monitor a spatial location and orientation of the instrument and aspatial location and orientation of the ultrasound transducer, acontroller communicatively coupled to the ultrasound transducer and theposition sensing system, and a display communicatively coupled to thecontroller, wherein the controller is configured to generate a twodimensional ultrasound image based on the ultrasound echo signals,display the ultrasound image on the display, determine a location of theultrasound image based on the spatial location and orientation of theultrasound transducer, determine an angle between the longitudinal axisof the instrument and the plane of the ultrasound image, generate on thedisplay a marker corresponding to a projection of at least a portion ofthe longitudinal axis of the instrument onto the plane of the ultrasoundimage based on the spatial location and orientation of the instrument,and indicate on the display the angle between the longitudinal axis ofthe instrument and the plane of the ultrasound image using a codedappearance characteristic of the marker corresponding to projection ofthe longitudinal axis of the instrument onto the ultrasound image.

In some embodiments according to this aspect, the controller isconfigured to determine a location of an axis-image plane intersectionof the longitudinal axis of the instrument and the plane of theultrasound image, and the portion of the longitudinal axis of theinstrument whose projection onto the ultrasound image is indicated bythe marker comprises the axis-image plane intersection. In someembodiments, the coded appearance characteristic of the marker comprisestwo lines meeting at a vertex and forming an angle corresponding to theangle between the longitudinal axis of the instrument and the plane ofthe ultrasound image, the angle of the marker bisected by an orthogonalprojection of the longitudinal axis of the instrument onto the plane ofthe ultrasound image, the vertex located at the axis-image planeintersection.

In some embodiments according to this aspect, the coded appearancecharacteristic of the marker comprises a coded luminosity indicative ofthe angle between the longitudinal axis of the instrument and the planeof the ultrasound image. In some embodiments according to this aspect,the coded appearance characteristic of the marker comprises a codedcolor indicative of the angle between the longitudinal axis of theinstrument and the plane of the ultrasound image.

In some embodiments according to this aspect, the controller isconfigured to determine a location of the tip of the instrument based onthe spatial location and orientation of the instrument, and to generateon the display a marker indicating the location of the tip of theinstrument.

In some embodiments according to this aspect, the controller isconfigured to register a reference position and indicate on the displaya marker corresponding to a projection of the reference position ontothe plane of the ultrasound image. In some such embodiments, theultrasound system comprises a user interface operable to register auser-indicated reference position, and the reference position comprisesthe user-indicated reference position registered by the user interface.

In some embodiments according to this aspect, the projection of thereference position onto the plane of the ultrasound image comprises anorthogonal projection onto the plane of the ultrasound image. In otherembodiments according to this aspect, the projection of the referenceposition onto the ultrasound image comprises a projection parallel tothe longitudinal axis of the instrument.

In some embodiments according to this aspect, the controller isconfigured to compute a distance between the reference position and theplane of the ultrasound image, and to indicate on the display thedistance between the reference position and the plane of the ultrasoundimage by applying a coded color and/or coded luminosity to the markercorresponding to the projection of the reference position onto the planeof the ultrasound image. In some embodiments according to this aspect,the controller is configured to indicate on the display the distancebetween the reference position and the plane of the ultrasound imageusing a coded marker forming part of the marker of the projection of thereference position onto the plane of the ultrasound image.

In some embodiments according to this aspect, the distance between thereference position and the plane of the ultrasound image comprises adistance along a line that is orthogonal to the plane of the ultrasoundimage. In other embodiments according to this aspect, the distancebetween the reference position and the plane of the ultrasound imagecomprises a distance along a line that is parallel to the longitudinalaxis of the instrument.

In some embodiments according to this aspect, the controller isconfigured to determine a location of an axis-reference position planeintersection of the longitudinal axis of the instrument with a planethat contains the reference position and is parallel to the plane of theultrasound image, and to generate on the display a marker correspondingto a projection of the axis-reference position plane intersection ontothe plane of the ultrasound image. In some such embodiments, theprojection of the axis-reference position plane intersection onto theplane of the ultrasound image comprises an orthogonal projection ontothe plane of the ultrasound image.

A further aspect of the invention provides a method for use ingenerating a display useful for directing a fine elongate instrument ina body, the instrument having a longitudinal axis and a tip, the methodcomprising receiving ultrasound echo signals returning from a portion ofthe body, determining a spatial location and orientation of theinstrument and a spatial location and orientation of the ultrasoundtransducer, generating a two dimensional ultrasound image based on theultrasound echo signals, displaying the ultrasound image on the display,determining a location of the ultrasound image based on the spatiallocation and orientation of the ultrasound transducer, determining anangle between the longitudinal axis of the instrument and the plane ofthe ultrasound image, generating on the display a marker correspondingto a projection of at least a portion of the longitudinal axis of theinstrument onto the plane of the ultrasound image based on the spatiallocation and orientation of the instrument, and indicating on thedisplay the angle between the longitudinal axis of the instrument andthe plane of the ultrasound image using a coded appearancecharacteristic of the marker corresponding to projection of thelongitudinal axis of the instrument onto the ultrasound image.

In some embodiments according to this aspect, the method comprisesdetermining a location of an axis-image plane intersection of thelongitudinal axis of the instrument and the plane of the ultrasoundimage, and the portion of the longitudinal axis of the instrument whoseprojection onto the ultrasound image is indicated by the markercomprises the axis-image plane intersection.

In some embodiments according to this aspect, the coded appearancecharacteristic of the marker comprises two lines meeting at a vertex andforming an angle corresponding to the angle between the longitudinalaxis of the instrument and the plane of the ultrasound image, the angleof the marker bisected by an orthogonal projection of the longitudinalaxis of the instrument onto the plane of the ultrasound image, thevertex located at the axis-image plane intersection.

In some embodiments according to this aspect, the coded appearancecharacteristic of the marker comprises a coded color and/or a codedluminosity indicative of the angle between the longitudinal axis of theinstrument and the plane of the ultrasound image.

In some embodiments according to this aspect, the method comprisesdetermining a location of the tip of the instrument based on the spatiallocation and orientation of the instrument, and generating on thedisplay a marker indicating the location of the tip of the instrument.

In some embodiments according to this aspect, the method comprisesregistering a reference position, and indicating on the display a markercorresponding to a projection of the reference position onto the planeof the ultrasound image. In some such embodiments, the method comprisesobtaining a user-indicated reference position via a user interface, andthe reference position comprises the user-indicated reference positionobtained via the user interface.

In some embodiments according to this aspect, the projection of thereference position onto the plane of the ultrasound image comprises anorthogonal projection onto the plane of the ultrasound image. In otherembodiments according to this aspect, the projection of the referenceposition onto the ultrasound image comprises a projection parallel tothe longitudinal axis of the instrument.

In some embodiments according to this aspect, the method comprisescomputing a distance between the reference position and the plane of theultrasound image, and indicating on the display the distance between thereference position and the plane of the ultrasound image by applying acoded color and/or coded luminosity to the marker corresponding to theprojection of the reference position onto the plane of the ultrasoundimage. In some embodiments according to this aspect, the methodcomprises computing a distance between the reference position and theplane of the ultrasound image, and indicating on the display thedistance between the reference position and the plane of the ultrasoundimage using a coded marker forming part of the marker of the projectionof the reference position onto the plane of the ultrasound image.

In some embodiments according to this aspect, the distance between thereference position and the plane of the ultrasound image comprises adistance along a line that is orthogonal to the plane of the ultrasoundimage. In other embodiments according to this aspect, the distancebetween the reference position and the plane of the ultrasound imagecomprises a distance along a line that is parallel to the longitudinalaxis of the instrument.

In some embodiments according to this aspect, the method comprisesdetermining a location of an axis-reference position plane intersectionof the longitudinal axis of the instrument with a plane that containsthe reference position and is parallel to the plane of the ultrasoundimage, and generating on the display a marker corresponding to aprojection of the axis-reference position plane intersection onto theplane of the ultrasound image. In some such embodiments, the projectionof the axis-reference position plane intersection onto the plane of theultrasound image comprises an orthogonal projection onto the plane ofthe ultrasound image.

Yet another aspect of the invention provide an ultrasound system for usein locating a reference position located in a body, the ultrasoundsystem comprising a memory operable to contain a spatial description ofthe reference position, an ultrasound probe operable to receiveultrasound echo signals returning from a portion of the body, anultrasound image processor communicatively coupled to the ultrasoundprobe, the ultrasound image processor operable to generate an ultrasoundimage based on the ultrasound echo signals, a position sensing systemoperable to determine a spatial location and orientation of theultrasound probe, an image plane locator communicatively coupled to theposition sensing system, the image plane locator operable to determine aspatial description of a plane of the ultrasound image based on thespatial location and orientation of the ultrasound probe, a geometrycomputer communicatively coupled to the image plane locator and thememory, the geometry computer operable to determine a spatialrelationship between the reference position and the plane of theultrasound image based on the spatial description of the referenceposition and the spatial description of the plane of the ultrasoundimage, a graphics processor communicatively coupled to the geometrycomputer, the marker generator operable to generate a marker indicativeof the spatial relationship between the reference position and the planeof the ultrasound image, and a display communicatively coupled to theultrasound image processor and the marker generator, the displayoperable to display the ultrasound image and the marker.

Yet a further aspect of the invention provides an ultrasound system foruse in guiding medical interventions in a body, the ultrasound systemcomprising an ultrasound transducer operable to receive ultrasound echosignals returning from a portion of the body, a fine elongate instrumentinsertable in the body, the instrument defining a longitudinal axis, aposition sensing system operable to monitor a spatial location andorientation of the instrument and a spatial location and orientation ofthe ultrasound transducer, a controller communicatively coupled to theultrasound transducer and the position sensing system; and a displaycommunicatively coupled to the controller, wherein the controller isconfigured to generate a two dimensional ultrasound image based on theultrasound echo signals, display the ultrasound image on the display,determine a location of the portion of the body depicted in theultrasound image based on the spatial location and orientation of theultrasound transducer, determine a first distance between a first linein the plane of the portion of the body depicted in the ultrasound imageand a first point along the longitudinal axis at which the longitudinalaxis of the instrument traverses the first line, determine a seconddistance between a second line in the plane of the portion of the bodydepicted in the ultrasound image and a second point along thelongitudinal axis at which the longitudinal axis of the instrumenttraverses the second line, generate on the display a first needle-imagealignment indicator having a first coded appearance characteristicindicative of the first distance, and generate on the display a secondneedle-image alignment indicator having a second coded appearancecharacteristic indicative of the second distance.

In some embodiments according to this aspect, the first line comprises afirst edge of the portion of the body depicted in the ultrasound imageand the second line comprises a second edge of the portion of the bodydepicted in the ultrasound image.

In some embodiments according to this aspect, the controller isconfigured to generate the first needle-image alignment indicator at afirst location on the display adjacent to a first line of the ultrasoundimage corresponding to the first line in the plane of the portion of thebody depicted in the ultrasound image, and to generate the secondneedle-image alignment indicator at a second location on the displayadjacent to a second line of the ultrasound image corresponding to thesecond line in the plane of the portion of the body depicted in theultrasound image. In embodiments according to this aspect, the first andsecond coded appearance characteristics may each comprise a color, afill pattern and/or a feature of shape. In some embodiments according tothis aspect, the first and second coded appearance characteristics areselected from a discrete set of different coded appearancecharacteristics. In some such embodiments, the controller is configuredto determine the first coded appearance characteristic by selecting itfrom the discrete set of different coded appearance characteristicsbased at least in part on whether the first distance is greater than afirst threshold distance, and to determine the second coded appearancecharacteristic by selecting it from the discrete set of different codedappearance characteristics based at least in part on whether the seconddistance is greater than a second threshold distance.

In some embodiments according to this aspect, the controller isconfigured to determine the first and second threshold distances basedat least in part on a maximum angular separation between thelongitudinal axis and the plane of the ultrasound image.

In some embodiments according to this aspect, the controller isconfigured to determine a first side of a plane of the portion of thebody depicted in the ultrasound image which the first point along thelongitudinal axis is on, determine a second side of the plane of theportion of the body depicted in the ultrasound image the second pointalong the longitudinal axis is on, determine the first coded appearancecharacteristic by selecting it from the discrete set of different codedappearance characteristics based on the first side, and to determine thesecond coded appearance characteristic by selecting it from the discreteset of different coded appearance characteristics based on the secondside.

In some embodiments according to this aspect, the first and second codedappearance characteristics of the respective first and secondneedle-image alignment indicators each comprise a combination of a colorand a feature of shape. In some such embodiments, the controller isconfigured to determine the color of the first needle-image alignmentindicator based at least in part on the first distance, determine thefeature of shape of the first needle-image alignment indicator based atleast in part on the first side, determine the color of the secondneedle-image alignment indicator based at least in part on the seconddistance, and determine the feature of shape of the second needle-imagealignment indicator based at least in part on the second side.

Still another aspect of the invention provides a method for providing adisplay for use in guiding a fine elongate instrument, the instrumentdefining a longitudinal axis, the method comprising receiving ultrasoundecho signals returning from a portion of the body, determining a spatiallocation and orientation of the instrument and a spatial location andorientation of the ultrasound transducer, generating a two dimensionalultrasound image based on the ultrasound echo signals, displaying theultrasound image on the display, determining a location of theultrasound image based on the spatial location and orientation of theultrasound transducer, determining a location of the portion of the bodydepicted in the ultrasound image based on the spatial location andorientation of the ultrasound transducer, determining a first distancebetween a first line in the plane of the portion of the body depicted inthe ultrasound image and a first point along the longitudinal axis atwhich the longitudinal axis traverses the first line, determining asecond distance between a second line in the plane of the portion of thebody depicted in the ultrasound image and a second point along thelongitudinal axis at which the longitudinal axis traverses the secondline, generating on the display a first needle-image alignment indicatorhaving a first coded appearance characteristic indicative of the firstdistance, and generating on the display a second needle-image alignmentindicator having a second coded appearance characteristic indicative ofthe second distance. In some embodiments according to this aspect, thefirst line comprises a first edge of the portion of the body depicted inthe ultrasound image and the second line comprises a second edge of theportion of the body depicted in the ultrasound image.

In some embodiments according to this aspect, the method comprisesgenerating the first needle-image alignment indicator at a firstlocation on the display adjacent to a first line of the ultrasound imagecorresponding to the first line in the plane of the portion of the bodydepicted in the ultrasound image, and generating the second needle-imagealignment indicator at a second location on the display adjacent to asecond line of the ultrasound image corresponding to the second line inthe plane of the portion of the body depicted in the ultrasound image.

In embodiments according to this aspect, the first and second codedappearance characteristics may each comprise a color, a fill pattern,and/or a feature of shape. In some embodiments according to this aspect,the first and second coded appearance characteristics are selected froma discrete set of different coded appearance characteristics. In somesuch embodiments, the method comprises determining the first codedappearance characteristic by selecting it from the discrete set ofdifferent coded appearance characteristics based at least in part onwhether the first distance is greater than a first threshold distance,and determining the second coded appearance characteristic by selectingit from the discrete set of different coded appearance characteristicsbased at least in part on whether the second distance is greater than asecond threshold distance.

In some embodiments according to this aspect, the method comprisesdetermining the first and second threshold distances based at least inpart on a maximum angular separation between the longitudinal axis andthe plane of the ultrasound image.

In some embodiments according to this aspect, the method comprisesdetermining a first side of a plane of the portion of the body depictedin the ultrasound image which the first point along the longitudinalaxis of the instrument is on, determining a second side of the plane ofthe portion of the body depicted in the ultrasound image which thesecond point along the longitudinal axis of the instrument is on,determining the first coded appearance characteristic by selecting itfrom the discrete set of different coded appearance characteristicsbased on the first side, and determining the second coded appearancecharacteristic by selecting it from the discrete set of different codedappearance characteristics based on the second side.

In some embodiments according to this aspect, determining each of thefirst and second coded appearance characteristics of the respectivefirst and second needle-image alignment indicators comprises determininga combination of a color and a feature of shape. Some such embodimentscomprise determining the color of the first needle-image alignmentindicator based at least in part on the first distance, determining thefeature of shape of the first needle-image alignment indicator based atleast in part on the first side, determining the color of the secondneedle-image alignment indicator based at least in part on the seconddistance, and determining the feature of shape of the secondneedle-image alignment indicator based at least in part on the secondside.

Still a further aspect of the invention provides an ultrasound systemcomprising an ultrasound transducer operable to receive ultrasound echosignals returning from a portion of the body, at least one firstposition marker connectable to the ultrasound transducer, at least onesecond position marker connectable to the instrument, a controllercommunicatively coupled to the ultrasound transducer and the first andsecond position markers, and a display communicatively coupled to thecontroller, wherein, when the at least one first position marker isconnected to the ultrasound transducer and the at least one secondposition marker is connected to the instrument, the controller isconfigured to monitor a spatial location and orientation of theultrasound transducer based on a first position signal from the at leastone first position marker, monitor a spatial location and orientation ofthe instrument based on a second position signal from the at least onesecond position marker, monitor a quality of the first position signal,monitor a quality of the second position signal, generate a twodimensional ultrasound image based on the ultrasound echo signals,determine a spatial location of the portion of the body depicted in theultrasound image based on the spatial location and orientation of theultrasound transducer, and to display on the display a plurality ofdisplay elements including the ultrasound image, and a markercorresponding to a projection of at least a portion of the longitudinalaxis of the instrument onto the plane of the ultrasound image based onthe spatial location and orientation of the instrument and the spatiallocation of the portion of the body depicted in the ultrasound image;and, when at least one of the quality of the first position signal isbelow a first quality threshold and the quality of the second positionsignal is below a second quality threshold, to generate an alert on thedisplay by doing one or more of inhibiting display of at least one ofthe plurality of display elements and changing an appearancecharacteristic of at least one of the plurality of display elements.

In some embodiments according to this aspect, the controller isconfigured to inhibit display of the ultrasound image when the qualityof the first position signal is below the first quality threshold. Insome embodiments according to this aspect, the controller is configuredto change an appearance characteristic of the ultrasound image when thequality of the first position signal is below the first qualitythreshold.

In some embodiments according to this aspect, the controller isconfigured to inhibit display of the marker when the quality of thesecond position signal is below the second quality threshold. In someembodiments according to this aspect, the controller is configured tochange an appearance characteristic of the marker when the quality ofthe second position signal is below the second quality threshold.

An aspect of the invention provides a method in an ultrasound systemcomprising receiving ultrasound echo signals returning from a portion ofthe body, monitoring a spatial location and orientation of an ultrasoundtransducer at which the ultrasound echo signals are received based on afirst position signal from at least one first position marker connectedto the ultrasound transducer, monitoring a spatial location andorientation of the instrument based on a second position signal from atleast one second position marker connected to the instrument, monitoringa quality of the first position signal, monitoring a quality of thesecond position signal, generating a two dimensional ultrasound imagebased on the ultrasound echo signals, determining a spatial location ofthe portion of the body depicted in the ultrasound image based on thespatial location and orientation of the ultrasound transducer,displaying a plurality of display elements including the ultrasoundimage and a marker corresponding to a projection of at least a portionof the longitudinal axis of the instrument onto the plane of theultrasound image based on the spatial location and orientation of theinstrument and the spatial location of the portion of the body depictedin the ultrasound image, and, when at least one of the quality of thefirst position signal is below a first quality threshold and the qualityof the second position signal is below a second quality threshold,generating an alert on the display by doing one or more of inhibitingdisplay of at least one of the plurality of display elements andchanging an appearance characteristic of at least one of the pluralityof display elements.

In some embodiments according to this aspect, the method comprisesinhibiting display of the ultrasound image when the quality of the firstposition signal is below the first quality threshold. In someembodiments according to this aspect, the method comprises changing anappearance characteristic of the ultrasound image when the quality ofthe first position signal is below the first quality threshold.

In some embodiments according to this aspect, the method comprisesinhibiting display of the marker when the quality of the second positionsignal is below the second quality threshold. In some embodimentsaccording to this aspect, the method comprises changing an appearancecharacteristic of the marker when the quality of the second positionsignal is below the second quality threshold.

In addition to the exemplary aspects and embodiments described above,further aspects and embodiments will become apparent by reference to thedrawings and by study of the following detailed descriptions.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings illustrate non-limiting embodiments.

FIG. 1 shows an example ultrasound probe and biopsy assembly as may beused with the invention.

FIG. 2 is a perspective view of a probe and a corresponding image plane.

FIG. 3 shows example ultrasound images.

FIG. 4A shows a display according to an example embodiment.

FIG. 4B is a perspective schematic illustration of an ultrasoundenvironment.

FIG. 5A is a side elevation schematic illustration of an ultrasoundenvironment.

FIG. 5B shows a display according to an example embodiment.

FIG. 6A is a side elevation schematic illustration of an ultrasoundenvironment.

FIG. 6B shows a display according to an example embodiment.

FIG. 7A is a top plan schematic illustration of an image.

FIG. 7B shows graphical illustrations of computations of in-planethreshold distances according to example embodiments.

FIG. 8A is a perspective schematic illustration of an ultrasoundenvironment.

FIG. 8B shows a graphical illustration of a computation of an in-planethreshold distance according to an example embodiment.

FIG. 9 shows a display according to an example embodiment.

FIG. 10 show a schematic diagram of an ultrasound operating environment.

FIG. 11 shows a schematic diagram of the ultrasound operatingenvironment depicted in FIG. 10.

FIG. 12 shows an ultrasound image display according to an exampleembodiment.

FIG. 13 shows an ultrasound image display according to an exampleembodiment.

FIG. 13A is a perspective view of a vacuum biopsy needle.

FIG. 13B shows an ultrasound image display according to an exampleembodiment.

FIG. 13C shows an ultrasound image display according to an exampleembodiment.

FIG. 14 shows an example shadow representation which could be displayedon a 2D monitor to indicate the 3D relationship of a needle to ananatomical structure.

FIG. 15 shows ultrasound equipment according to an embodiment thatprovides automatic steering of a plane in which ultrasound images areacquired.

FIG. 16 is a flow diagram of a method for generating a 3D model.

FIG. 17 shows an example system which can generate a 3D model fromultrasound frames.

FIG. 18 is a block diagram of apparatus according to an exampleembodiment that includes a biopsy assembly, an ultrasound probe, and a3D position sensing system.

FIG. 19 shows an example image that may be displayed on a monitor.

FIG. 20 shows a method according to one example embodiment.

FIG. 21 shows an example system in which a position sensing systemdetermines positions of an ultrasound transducer and a needle and aninstruction generator generates instructions to assist a user.

FIG. 22 shows an ultrasound imaging probe having a built-in positionsensor.

FIG. 23 is a side elevation cross-sectional view of a marker positioningapparatus according to an example embodiment.

FIG. 24 is a top plan view of a marker positioning apparatus accordingto an example embodiment.

FIG. 25 shows a real-time display of marker location and orientationaccording to an example embodiment.

FIG. 26 is a perspective view of an example marker.

FIG. 27 shows schematically the use of two probe positions to define avolume for a 3D model.

FIG. 28 illustrates one way to convert between coordinates of a pixel ina single ultrasound frame and the coordinate system for a 3D model.

FIG. 29 illustrates an embodiment involving placement of a needle in awoman's breast.

FIG. 30 illustrates an application involving placing an epiduralcatheter.

FIG. 31 shows an ultrasound system according to an example embodiment.

FIG. 32 shows an ultrasound system according to an example embodiment.

FIG. 33 shows an ultrasound system according to an example embodiment.

DESCRIPTION

Throughout the following description, specific details are set forth inorder to provide a more thorough understanding of the invention.However, the invention may be practiced without these particulars. Inother instances, well known elements have not been shown or described indetail to avoid unnecessarily obscuring the invention. Accordingly, thespecification and drawings are to be regarded in an illustrative, ratherthan a restrictive, sense.

FIG. 1 shows an ultrasound probe 12. Probe 12 comprises a transducerarray 14 that can generate high frequency vibrations and transmit thosehigh frequency vibrations into the body of a patient P. The vibrationsare reflected from various structures and interfaces within patient P.Reflected signals are detected at transducer array 14 where they areconverted to electronic form and delivered to an ultrasound system (notshown in FIG. 1) for further analysis. Transducer array 14 may comprisea one or two-dimensional array of transducer elements, for example. Theparticular arrangement of transducer elements in array 14 may beselected based upon the medical application for which the probe 12 willbe used.

To create a diagnostic image, an ultrasound controller causes electricalexcitation signals to be delivered to elements of transducer array 14.The transducer elements convert the excitation signals into ultrasonicvibrations. The ultrasonic vibrations typically have frequencies in therange of about 2 megahertz to about 15 megahertz. This is not mandatory.Embodiments may employ frequencies outside of this range.

The ultrasonic vibrations are scattered and/or reflected by variousstructures in the patient's body. Some of the reflected and/or scatteredultrasonic vibrations, which may be called echos, are received attransducer array 14. The distance from the transducer array 14 to aparticular location at which echos are generated may be determined bythe time between the transmission of an ultrasonic vibration and thereceipt of an echo of that ultrasonic vibration at transducer array 14.The direction relative to probe 12 of a location at which an echo isgenerated may be determined by processing the echo signals. Various beamforming techniques may be used to determine the directions from whichechos arrive at transducer array 14.

For example, in so-called B-mode imaging, a 2D image of a selectedcross-section of the patient's body is generated. Because the positionand orientation of transducer array 14 is fixed in probe 12, theparticular cross section represented by an ultrasound image depends uponthe current position and orientation of probe 12 relative to thepatient's body. Moving probe 12 relative to the patient's body willresult in a different cross section being imaged.

FIG. 1 shows two scattering locations, P1 and P2. P1 is located atposition R1, θ1. P2 is at location R2, θ2. These locations are bothdetermined with reference to a coordinate system that can be consideredto be attached to probe 12.

The position and orientation of probe 12 are monitored by a 3D positionsensing system 16. The 3D position sensing system 16 may include one ormore base units and one or more markers carried on probe 12. In theillustrated embodiment, probe 12 includes a plurality of positionmarkers 15. In the illustrated embodiment, there are three positionmarkers, 15A, 15B, and 15C. Position markers 15A, 15B, and 15C are notlocated along a common line. Therefore, if the locations of positionmarkers 15A, 15B, and 15C are known, the position and orientation inspace of probe 12 is uniquely determined. Since the particular crosssection represented by an ultrasound image depends upon the currentposition and orientation of probe 12, the position and orientation ofultrasound images can be determined from the position and orientation inspace of probe 12.

The positions of location markers 15 relative to a global coordinatesystem are measured by 3D position sensing system 16. In the illustratedembodiment the sensor system includes a position base unit 17. 3Dposition base unit 17 and position markers 15 may comprise any suitabletechnology. For example, 3D position base unit 17 may detectelectromagnetic or other fields emitted by position markers 15 or viceversa. In some embodiments the position base unit 17 generates amagnetic field that is sensed by position markers 15. A 3D positionsensing system may, for example, comprise a medSAFE™ or drive BAY™position sensor available from Ascension Technology corporation ofBurlington, Vt., USA.

Some 3D position sensing technologies permit both the location andorientation of a single position marker to be determined. Where such 3Dposition sensing technologies are used, fewer position markers 15 arerequired to determine the location and orientation of probe 12 thanwould be the case for position markers for which only position isdetermined. For example a single 6 degree of freedom position marker maybe used in a compatible position sensor to obtain both position andorientation information for a probe 12. Even in embodiments which detectthe orientations of position markers, some redundant position markers 15may be provided. In embodiments which provide more position markers thanare required to identify position and orientation of probe 12, positionsof the additional position markers may be monitored by 3D position baseunit 17 and used to provide information regarding the position andorientation of probe 12 of enhanced accuracy.

FIG. 1 also shows a biopsy apparatus 19 which includes a handle 20 and aneedle 21. Biopsy apparatus 19 includes one or more position markers 15.In the illustrated embodiment, there are two position markers 15,individually identified as 15D and 15E. In the illustrated embodiment,position markers 15D and 15E are located so that they correspond toreference positions on an extension of a longitudinal axis of needle 21.Neglecting rotations about the axis of needle 21, the position andorientation of needle 21 can be uniquely determined if the positions ofposition markers 15D and 15E are known. In the illustrated embodiment,the reference positions of location markers 15D and 15E are monitored by3D position sensing system 16.

In an alternative embodiment, biopsy apparatus 19 has one positionmarker of a type such that position base unit 17 can determine both aposition and orientation of the position marker. The one position markermay, for example, comprise a six degrees of freedom marker. Additionalposition markers may optionally be provided on biopsy apparatus 19.

In the illustrated embodiment, position markers 15D and 15E are builtinto a handle of biopsy apparatus 19. Needle 21 is detachably affixableto the handle.

In some embodiments, position markers are built into probe 12, biopsyapparatus 19 and/or needle 21, such as in the manner described inapplication Ser. No. 12/703,706 filed on 10 Feb. 2010 and entitledULTRASOUND SYSTEMS INCORPORATING SPATIAL POSITION SENSORS AND ASSOCIATEDMETHODS, which is hereby incorporated herein by reference.

It can be appreciated that the apparatus illustrated in FIG. 1 mayfacilitate the placing of needle 21 into the body of patient P such thatneedle 21 may be used to acquire a tissue sample or place something at adesired location within patient P. Specifically, when an ultrasoundimage 23 is generated from ultrasound data acquired by probe 12, theprecise location and orientation of needle 21 relative to thatultrasound image can be determined from the known locations of positionmarkers 15 on probe 12 and biopsy assembly 19. Having this informationallows the location of needle 21 to be illustrated clearly on image 23(even if the ultrasound echos do not provide a clear image of needle21). In the illustrated embodiment, needle 21 is represented by acomputer-generated line 24 that shows the position of needle 21 in image23, as calculated based on the relative positions of position markers15.

In some embodiments, needle 21 is detachable from handle 19. In suchembodiments, needle 21 may be connected to handle 19 using a couplingwhich fixes the orientation of needle 21 relative to handle 19 such thatthe axis of needle 21 will have a predetermined alignment with theposition markers 15D and 15E. Where a replaceable needle 21 is providedthen there is a possibility that needles 21 of different lengths may beused. Advantageously, a procedure is provided for calibrating theapparatus to indicate the location of the tip of needle 21 relative toposition marker(s) 15D and 15E.

In one embodiment, a target 25 that has a location known to 3D positionsensing system 16 is provided. For example, target 25 may be provided onprobe 12 or on 3D position base unit 17. In some embodiments target 25is provided on another apparatus having a known position. A user cantouch the tip of needle 21 to target 25 (or 25A). The user can indicatewhen this has been done by using a control (for example by pressing abutton on biopsy assembly 19 or probe 12 or on a connected ultrasoundsystem) or by automatically detecting that needle 21 has contactedtarget 25 or 25A. For example, the contact of needle 21 with a target 25may be detected by providing a pressure sensor in conjunction withtarget 25 or 25A that detects the pressure exerted by needle 21 on thetarget, an electrical sensor that senses an electrical signal deliveredto the target by way of needle 21, a capacitive sensor, or the like.

Since needle 21 is used in a surgical procedure it is generallyimportant that needle 21 be sterile and kept sterile. Target 25 shouldbe also sterile. In some embodiments, target 25 is provided on a sterilecover that locks in place over probe 12. In some embodiments, target 25includes a marking on probe 12 that can be seen through a transparentportion of the cover. Other embodiments may also be contemplated.

In another embodiment, calibrating the apparatus to indicate thelocation of the tip of needle 21 relative to one or more positionmarkers 15D and 15E may be achieved without the use of a target. Tocalibrate the needle, a user moves the body of the needle while keepingthe position of the needle tip unchanged. This can be done, for example,by placing the tip of the needle on any surface and moving the handlearound. The positions of position markers 15D and 15E are monitored by3D position sensing system 16 and recorded as the needle is rotated.Since the tip of the needle remains stationary throughout, its locationin the global coordinate system (i.e., its position relative to positionbase unit 17) is fixed. The distance between each of position markers15D and 15E and the needle tip is also fixed. Consequently, positionmarkers 15D and 15E each move on the surface of a spherical shell.

The position of the needle tip relative to position markers 15D and 15Emay be determined using an algorithm that applies the knowledge that themeasured positions of the position markers are on spherical shells(which in this example are centered on the stationary tip of theneedle). The algorithm may use the fact that many position measurementscan be obtained to reduce the effect of uncertainties in the positionsmeasured for position markers 15D and 15E. For example, a recursiveleast squares position determination algorithm may be employed toestimate the offset between position markers 15D and 15E and the needletip. An example of such an algorithm is explained in P. R. Detmer, G.Basheim, T. Hodges, K. W. Beach, E. P. Filer, D. H. Burns and D. E.Strandness Jr, 3D ultrasonic image feature localization based onmagnetic scanhead tracking: in vitro calibration and validation,Ultrasound Med. Biol. 20, 923-936 (1994) which is hereby incorporated byreference for all purposes. After determining offset, the needle tipposition in the global coordinate system can be estimated in real-timethrough the position and orientation information provided from theposition markers 15D and 15E.

Because position and orientation information from position markers onprobe 12 indicates the position of the markers in 3D space, locating theultrasound image (scan plane) in 3D space requires determining therelationship between the position and orientation of the ultrasoundimage and the markers on probe 12.

Various methods for calibrating freehand ultrasound probes that may beapplied to establish a correspondence between points in an ultrasoundimage obtained using probe 12 and corresponding points in a globalcoordinate system are described in Freehand 3D Ultrasound Calibration: AReview, P-W. Hsu, R. W. Prager A. H. Gee and G. M. TreeceCUED/F-INFENG/TR 584, University of Cambridge Department of Engineering,December 2007 which is hereby incorporated herein by reference.

One example method for determining a transform that is characteristic ofthis relationship involves moving and/or rotating probe 12 to acquireimages of a point phantom (e.g. the center of a ball phantom, a stringphantom, or the like) from a variety different positions and angles. Theposition of the fixed feature in each ultrasound image, as measuredrelative to the ultrasound image coordinate system, is recorded alongwith the position and orientation of probe 12 in the global coordinatesystem, which is determined from position markers 15A, 15B and 15C. Ifthe location of the point phantom in the global coordinate system isassumed to correspond to the origin of a phantom coordinate system F,then the relationship between the location (u,v) of the point phantom inthe ultrasound image and that its location at the origin of the phantomcoordinate system can be expressed as

$\begin{matrix}{{T_{F\leftarrow W}T_{W\leftarrow M}T_{M\leftarrow I}{T_{s}\begin{pmatrix}u \\v \\0\end{pmatrix}}} = \begin{pmatrix}0 \\0 \\0\end{pmatrix}} & (1)\end{matrix}$where T_(F←W) is the transform from the global coordinate space to thephantom coordinate space, T_(W←M) is the transform from the markercoordinate space to the global coordinate space and T_(M←I) is thetransform from the image coordinate space to the marker coordinate spaceand T_(s) is a scaling matrix that accounts for the differences in unitsof the ultrasound image and the real world (e.g., pixels versusmillimeters). Since the position and orientation of the position markersis read from position sensing system 16, T_(W←M) is known for each imageacquired. Because the orientation of an arbitrary coordinate systemrelative to the global coordinate system is not relevant, the rotationsof T_(F←W) can be set to arbitrary values (for example zero). Thisleaves an equation with 11 unknowns: the two unit scaling factors (onefor each dimension of the ultrasound image), six calibration parameters(3 translations and 3 rotations from the ultrasound image orientationand position to the marker orientation and position) and threetranslations for the shift of the point phantom to the origin of thephantom coordinate system. If N images of the point phantom are obtainedfrom a diversity of positions and orientations, the 11 unknowns can befound, for example, by minimizing

$\begin{matrix}{f = {\sum\limits_{i = 1}^{N}{{T_{F\leftarrow W}T_{W\leftarrow M_{i}}T_{M\leftarrow I}{T_{s}\begin{pmatrix}u_{i} \\v_{i} \\0\end{pmatrix}}}}}} & (2)\end{matrix}$

The function of Equation (2) can be minimized using a recursive leastsquares algorithm, such as the Levenberg-Marquardt algorithm. Afterfinding the transform matrix T_(M←I), any point on the ultrasound imagecan be first transformed into the marker coordinate system then into theglobal coordinate system using the position and orientation informationprovided from the position sensor system 16.

Finding the optimized transform matrix for the ultrasound probe (theprobe calibration process) can be simplified if the position andorientation of the sensor (marker) inside the probe are known to areasonable degree of accuracy from the start. The following is anexample of a simplified probe calibration method which applies apreviously calibrated needle. Ideally, if the coordinate axes of theprobe marker(s) are aligned with coordinate axes of the probe a rotationmatrix is not needed. In this case calibration would only involvedetermining offsets between the ultrasound image orientation andposition to the marker orientation and position. However, in thereal-case it is difficult to perfectly align the axes of a marker withthose of a probe. In a particular case, each axis might be off by a fewdegrees.

To correct for misalignment of the axes of the marker and probe(transducer array) one can perform a calibration process to arrive at atransformation that will rotate and translate the coordinates in theultrasound image to corresponding coordinates in 3D space. One exampleway to achieve this is to separate the rotation matrix into threematrixes targeting each axis and calibrate the axes one-by-one. Such aprocess can be done automatically or manually. In an example automaticmethod, one can begin by imaging a needle with different probe anglesand directions. Because the needle has been previously calibrated itsposition and orientation in space are known. Because any misalignmentsare reasonably small an approximate transformation is also known. Foreach angle, one can determine the distance between the actualintersection of the needle and the ultrasound imaging plane asidentified by locating the needle in the ultrasound data with thepredicted intersection of the needle and the ultrasound imaging plane.The predicted intersection may be determined from the known position ofthe needle, the known position of the probe marker(s) and theapproximate transformation. A best-fit algorithm such as a recursiveleast squares position determination algorithm can then be used to findthe best offset and rotation to minimize the total square error. A bruteforce search minimizing the error can also be used.

In an example manual optimization method, the rotation and translationbetween each axis is calibrated one-by-one. FIG. 2 is a perspective viewof a probe 210 scanning an image plane 211. The axis definitions in thefollowing explanation correspond to labels shown for axis lines 218. TheX axis extends in a direction away from the face of probe 210 (i.e., outof the plane of the transducer array and into the depth of the imageplane). The Y axis extends along the long edge of the face of probe 210(i.e., along the long dimension of the transducer array; laterallyacross the image plane). The Z axis extends along the short edge of theface of probe 210 (i.e., along the short dimension of the transducerarray; out of the image plane). The ultrasound image corresponds to aplanar slice defined between two planes parallel to the plane defined bythe X and Y axes. The transducer array lies in a plane defined by the Yand Z axes.

To calibrate the X axis, the needle may be placed in a position parallelto the long edge 210A of probe 210. The X axis rotation angle isadjusted until the surface angle 212 is 0 or 180 degrees. To calibratethe Y axis, the needle may be put into a position parallel to the shortedge 210B of the probe head 210 and the Y axis rotation angle adjusteduntil the out-plane angle 214 and surface angle 212 are 90 degrees. Tocalibrate the Z axis, the needle may be put into a position parallel tothe long edge 210A of probe 210 and the Z axis rotation angle adjusteduntil the in-plane angle 216 is 0 or 180 degrees. The positioning of theneedle may be assisted by locating the needle in the ultrasound image.In some embodiments, the location of the needle in the ultrasound imageis determined automatically, such as, for example, by patternrecognition algorithms. The manual calibration process may use asuitable apparatus for positioning the probe and/or needle.

Because the axes of the marker(s) may not be aligned with the ultrasoundimage axes, rotating a marker plane about one marker axis may correspondto rotation about more than one ultrasound image axis. To overcome thisdifficulty, the marker planes can be rotated through ultrasound imageaxes instead. This can be achieved by right multiplying an axisadjustment matrix before and after the axis rotation. The right axisadjustment matrix can be estimated by putting the needle parallel to theprobe surface and adjusting rotation angles. For example, if rotatingthe needle about only the Z axis changes the in-plane angle but not theout-plane angle and surface angle, then the marker Z axis is alignedwell with the probe Z axis.

A simple way to verify the accuracy of the needle calibration is torotate the needle about its axis 360 degrees (with, for example, 30degree as the step size) while keeping the position of the probe fixed.The intersection of the needle line with the B-mode image (indicated bythe cross mark) should not change by more than a small amount e.g. 1 mm.

The offsets and rotations required to calibrate a probe or needle may bestored in an electronic data file. In some embodiments, electronic datafiles are structured according to the XML format. In some embodiments, auser changing probes or needles can load calibration information for theparticular newly attached probe or needle from an electronic data file.This saves the user having to calibrate the newly attached probe orneedle. In some embodiments the data file is stored in a memory on theprobe or a connector for the probe such that the calibration informationis always available to an ultrasound machine to which the probe isconnected.

Having knowledge of the location of needle 21 relative to the plane atwhich an ultrasound image 23 is obtained can permit the calculation anddisplay of images and other feedback that helps a user to visualize therelative locations of needle 21 and a targeted abnormality or otherlocation within a patient P. FIG. 3 shows some examples. In FIG. 3, amain ultrasound image 23 shows an image 29 of some anatomical structureinto which it is desired to place needle 21. A plurality of graphicalelements, namely line 24, line 30 and marker 32 are superposed on image23. Line 24 represents a projection of needle 21 into the plane of thebody represented in image 23. Line 30 superposed on image 23 indicatesthe path that will be followed by the tip of needle 21 as it continuesto be inserted. Marker 32 is superposed on image 23 at the locationcorresponding to the intersection of needle 21 with the plane and thebody represented by image 23.

In some embodiments, graphical elements, such as, for example, markers,lines and the like, may comprise translucent and/or stippled elements toallow the features of an ultrasound image to be made out by a user whenthe graphical elements are superposed on the image. In some embodiments,the display of a graphical element may be inhibited and/or theappearance of the graphical element modified to allow the features of anultrasound image beneath the element to be made out, upon the detectionof a condition of the ultrasound environment. For example, in someembodiments, when the tip of needle 21 is in or almost in the plane ofthe ultrasound image, marker 32 is modified or removed so that the usercan see in image 23 the tip of needle 21 (or a line representative ofthe tip of needle 21) approaching the target location.

In some embodiments, marker 32, line 24 and/or line 30 may comprise becoded by luminosity (brightness), color, size, shape, linestyle or otherappearance characteristic to indicate a feature or aspect of anultrasound environment, such as, for example, the distance betweenmarker 32 and the tip of needle 21. For example, marker 32 may bedisplayed in a color that transitions along a spectrum from green to redwhen the tip of the needle 21 nears the plane of image 23. In someembodiments, marker 32 comprises a circle, the diameter of the circlevarying according to the distance between the tip of the needle and theplane of the image). Some embodiments comprise a scale that relates acoded appearance characteristic of marker 32, line 24 and/or line 30 todistance between the tip of needle 21 and marker 32.

In some embodiments, an appearance characteristic of marker 32, line 24and/or line 30 is coded according to which side of the image plane thetip of needle 21 is on. For example, if needle 21 has not passed throughthe plane of an image marker 32 may be shown in blue, but if needle 21has passed through the plane of the image, then marker 32 may be shownin yellow. In some embodiments, as the distance between the tip ofneedle 21 and a first side of the plane of image 23 closes, the color inwhich marker 32 is shown changes along a first spectrum (e.g., aspectrum running from green to blue), and as the distance between thetip of needle 21 and a second side of the plane of image 23 closes, thecolor in which marker 32 is shown changes along a second spectrum (e.g.,a spectrum running from red to yellow). Some embodiments comprise ascale that relates a coded appearance characteristic of marker 32, line24 and/or line 30 to distance between the tip of needle 21 and a side ofthe plane of image 32.

It may be possible for needle 21 to be oriented such that its trajectorydoes not intersect the plane of image 23 in the field of view of image23. In some embodiments, a coded appearance characteristic of line 30and/or line 24 indicates the fact that the intersection of thetrajectory of needle 21 with the plane of image 23 does not lie in thefield of view of image 23. For example, when the intersection of theaxis of needle 21 with the plane of image 23 does not lie in the fieldof view of image 23, line 30 may be shown in a different color (e.g.,gray) as compared with its color when the intersection of the trajectoryof needle 21 with the plane of image 23 does lie in the field of view ofimage 23. Other aspects of the display may indicate the fact that theintersection of the trajectory of needle 21 with the plane of image 23does not lie in the field of view of image 23. For example, image 23could be shown with a different range of colors or luminances (e.g.,image 23 could be made gray or darker) if the intersection does not liein the field of view of image 23.

In some embodiments, a coded appearance characteristic of marker 32,line 24 and/or line 30 indicates the angle that the trajectory of needle21 makes with the plane of image 23. For example, the color used todisplay line 30 may traverse a spectrum of colors as needle 21 movesfrom parallel to the plane of image 23 to perpendicular to the plane ofimage 23. In such embodiments, the projection of needle 21 and itstrajectory onto image 23 as lines 24 and 30 indicates the orientation ofneedle 21 in a first plane, and the brightness of line 30 indicates theorientation of needle 21 in a second plane orthogonal to the first.Accordingly, in such embodiments, a user may be able to ascertain, atleast approximately, the orientation of needle 21 relative to featuresin the plane of image 23 in three dimensions from only line 30.

FIG. 4A shows a display 330 according to an example embodiment. FIG. 4Bis a perspective schematic illustration of an ultrasound environment350. In environment 350, trajectories 354 and 364 intersect plane 352,respectively, at intersections 356 and 366. Trajectories 354 and 364intersect plane 352, respectively, at oblique angles 358 and 368. Angle368 is larger than angle 358. In display 330, image area 332 correspondsto plane 352. Lines 334 and 344 represent, respectively, the orthogonalprojections of trajectories 354 and 364 onto plane 352. Coded markers336 and 346 represent, respectively, intersections 356 and 366. Codedmarker 336 comprises sector 336A. The acute angle between the radii thatdefine sector 336A corresponds to angle 358. Sector 336A is bisected bythe projection of trajectory 344 onto plane 352. Coded marker 346comprises sector 346A. The acute angle between the radii that definesector 346A corresponds to angle 368. Sector 346A is bisected by theprojection of trajectory 364 onto plane 352.

In an ultrasound systems comprising display 330, lines 334 and 344indicate the orientations of trajectories 354 and 364, respectively, inplane 352, and sectors 336A and 346A indicate the orientations,respectively, trajectories in planes orthogonal to plane 352.Advantageously, a user may be able to ascertain, at least approximately,the orientation of trajectory 334 relative to features in the plane ofimage area 332 in three dimensions from line 334 and coded marker 336.It will be appreciated that there exist many variations on theconfigurations of coded markers 334 and 346 that would convey similarinformation regarding the orientation of trajectories 354 and 364 inplanes orthogonal to plane 352. For example, coded markers 354 and 364could comprise only the radii defining the sectors (thus appearing asrays of the angles whose vertexes lie, respectively, at the imagelocations corresponding to the intersection of trajectories 354 and 364with plane 352 and which are bisected, respectively, the orthogonalprojections of trajectories 354 and 364 onto plane 352). In otherembodiments, coded appearance characteristics of coded markers, such assize, color, intensity, shape, linestyle, of the like, may be used toindicate the orientations of trajectories in planes orthogonal to theimage plane.

In some embodiments, the angle that an instrument makes with the planeof an ultrasound image is indicated by coded markers whose appearancesare related, respectively, to distances between the axis defined by theinstrument and two or more lines in the plane of the image that areintersected by the projection of the axis onto the plane of the image.For example, the angle that an instrument makes with the plane of anultrasound image may be indicated by coded markers whose appearances arerelated, respectively, to the distances between the axis defined by theinstrument and lines at, along or near two or more edges of the image.

FIGS. 5A and 6A show schematic side elevation views of ultrasoundenvironments 380 and 390, respectively. FIGS. 5B and 6B show exampledisplays 386 and 396, respectively. Displays 386 and 396 depict aspectsof environments 380 and 390, respectively. In environment 380 a needle382 is at an angle to a middle plane 383 of an ultrasound image slice384. Needle 382 crosses middle plane 383 and tip 382A lies in slice 384.A trajectory 382B of needle 382 traverses a first edge 384A of slice384. Where trajectory 382B traverses edge 384A of slice 384, trajectory382B is in slice 384 and at a distance 385A from middle plane 383.Needle 382 traverses a second edge 384B of slice 384. Where needle 382traverses edge 384B of slice 384, needle 382 is in slice 384 and at adistance 385B from middle plane 383.

In environment 390 a needle 392 is at an angle to a middle plane 393 ofan ultrasound image slice 394. Needle 392 crosses middle plane 393 andtip 392A lies in slice 394. A trajectory 392B of needle 392 traverses afirst edge 394A of slice 394. Where trajectory 392B traverses edge 394Aof slice 394, trajectory 392B is in slice 394 and at a distance 395Afrom middle plane 393. Needle 392 is out of slice 394 where needle 392traverses edge 394B of slice 394. Where needle 392 traverses edge 394Bof slice 394, needle 392 is at a distance 395B from middle plane 393.

In FIG. 5B, display 386 comprises image area 387. Image area 387 isbounded by image edges, including opposite image edges 387A and 387B,which correspond, respectively, to edges 384A and 384B of slice 384.Image area 387 comprises an image depicting the region in slice 384.Image area 387 comprises a computer-generated line 388 indicative of theprojection of needle 382 onto the plane of the image depicted in imagearea 387, and a stippled line 388A indicative of the projection oftrajectory 382B onto the plane of the image depicted in image area 387.Display 386 comprises needle-image alignment indicators 389A and 389B.

In the illustrated embodiment, needle-image alignment indicators 389Aand 389B are in the shape of elongated bars whose long edges areparallel to the lateral edges of image area 387. Indicators 389A and389B have a coded fill appearance indicative of proximity between needle382 and/or its trajectory 382B and middle plane 383 of image slice 384.A coded fill appearance may comprise, for example, a color, a pattern, acombination thereof, or the like. In some embodiments, needle-imagealignment indicators indicate proximity between a needle axis and animage slice using coded appearance characteristics of color, pattern,shape, size, linestyle or the like.

The coded fill appearance of indicators 389A and 389B indicate thatneedle 382 and its trajectory 382B are close to middle plane 383 wherethey traverse edges 384A and 384B of slice 384. Since both indicators389A and 389B indicate that needle 382 and its trajectory 382B are closeto middle plane 383 where they traverse edges 384A and 384B of slice384, a user may infer that the axis defined by needle 382 (i.e., needle382 and its trajectory 382A) is substantially parallel to and nearmiddle plane 383.

In FIG. 6B, display 396 comprises image area 397. Image area 397comprises an image depicting the region in slice 394. Image area 397 isbounded by image edges, including opposite image edges 397A and 397B,which correspond respectively, to edges 394A and 394B of slice 394.Image area 397 comprises a computer-generated line 398 indicative of theprojection of needle 392 onto the plane of the image depicted in imagearea 397, and a stippled line 398A indicative of the projection oftrajectory 392B onto the plane of the image depicted in image area 397.Display 396 comprises needle-image alignment indicators 399A and 399B.

In the illustrated embodiment, needle-image alignment indicators 399Aand 399B are in the shape of elongated bars whose long edges areparallel to the lateral edges of image area 397. Indicators 399A and399B have a coded fill appearance indicative of proximity between needle392 and/or its trajectory 392B and middle plane 393 of image slice 394.Indicators 399A and 399B have different coded fill appearances. Thecoded fill appearance of indicator 399A indicates that trajectory 392Bis close to middle plane 393 where it traverses edge 394A of slice 394.The coded fill appearance of indicator 399B indicates that needle 392 isfar from middle plane 393 where needle 392 traverses edge 394B of slice394. Since indicator 399A indicates trajectory 392B is close to middleplane 393 where it traverses edge 394A of slice 394 and indicator 399Bindicates that needle 392 is far from middle plane 393 where ittraverses edge 394B of slice 394, a user may infer that the axis definedby needle 382 is not substantially parallel to slice 384.

It will be appreciated that the angle that an instrument makes with theplane of an ultrasound image may be indicated by coded markers whoseappearance characteristics are related, respectively, to the distancesbetween the axis defined by the instrument and lines in the plane of theimage that are intersected by the projection of the axis onto the planeof the image and that themselves intersect. For example, the appearancecharacteristics of needle-image alignment indicators may be related,respectively, to the distances between the axis defined by theinstrument and two edges of an image slice that share a vertex. In someembodiment where the projection of the axis defined by a needle onto animage “cuts the corner” of the image, needle-image alignment indicatorsmay be provided along the edges that form the corner that is cut.

Those skilled in the art will understand that the methods and apparatusdisclosed herein for generating needle-image alignment indicators forcircumstances where a needle axis traverses opposite image edges may begeneralized for application to circumstances where a needle axistraverses lines that are not parallel (e.g., lines that intersect eachother), such as, for example, lines at, along or adjacent to oppositeedges of an image that are not parallel (e.g., as in generallytrapezoidal ultrasound images acquired by curved transducer arrays). Itwill further be appreciated that methods and apparatus disclosed hereinfor generating needle-image alignment indicators may be applied usinglines intersected by the projection of an axis defined by a needle,which lines are not straight, such as, for example, lines comprisingimage edges that comprise curves, arcs and the like.

The coded appearance characteristics of needle-image alignmentindicators, such as indicators 389A, 389B, 399A and 399B may indicateproximity over a continuous range, or may indicate discrete ranges ofproximity. In some embodiments, a needle-image alignment indicatorcomprises an appearance characteristic from a continuous range ofappearance characteristics which corresponds to a range of distancesbetween a middle plane of an image slice and the point at which theneedle axis traverses an edge of the image slice. For example, aneedle-image alignment indicator may comprise a color from a continuousspectrum of color which corresponds to a range of distances between amiddle plane of an image slice and the point at which a needletrajectory traverses an edge of the image slice.

In some embodiments, a needle-image alignment indicator comprises anappearance characteristic from a discrete set of appearancecharacteristics, each of which corresponds to a distinct range ofdistances between a middle plane of an image slice and the point atwhich a needle axis traverses an edge of the image slice. For example, acoded appearance characteristic may comprise one of a set of two colors(e.g., green and red), with green corresponding to distances between amiddle plane of an image slice and the point at a needle axis traversesan edge of the image slice of less than or equal to a pre-determinedthreshold distance and with red corresponding to distances greater thanthe predetermined threshold distance. In embodiments where a codedappearance characteristic of a needle-image alignment indicator is drawnfrom a set of different appearance characteristics (e.g., colors,patterns, symbols, combinations thereof, or the like), one of which isindicative of close proximity to and/or presence within an ultrasoundimage slice, a user may use the indicators to quickly determine whetherthe needle axis is in-plane, partially in-plane or out-of-plane. Such ameans for quickly determining whether the needle axis is in-plane,partially in-plane, or out-of-plane may be useful where a user isattempting to advance a needle in the plane of an ultrasound image.

In some embodiments, a coded appearance characteristic of a needle-imagealignment indicator comprises an appearance characteristic that makesthe indicator indistinguishable from the background on which it isdisplayed when the distance between a needle and/or its trajectory isgreater than a threshold distance from a point in an image plane. Inother words, a needle-image alignment indicator may only appear to auser when a needle and/or its trajectory is sufficiently proximate to animage plane.

In some embodiments, a needle-image alignment indicator may indicate aside of the middle plane of the image slice which the needle axis iswhere it traverses an edge of the image slice. For example, aneedle-image alignment indicator may comprise a coded appearancecharacteristic of a feature of shape selected from a set of shapes. Sucha coded appearance characteristic may be selected, for example, from aset of symbols [+, −, =], in which the ‘+’ symbol indicates that theneedle/trajectory-image edge crossing is more than a pre-determineddistance away from the middle plane of the image on a first side of theimage, ‘−’ symbol indicates that the needle/trajectory-image edgecrossing is more than the pre-determined distance away from the middleplane of the image on a second side of the image, and the ‘=’ symbolindicates that the needle/trajectory-image edge crossing is within thepre-determined distance from the middle plane of the image. In someembodiments, a needle-image alignment indicator comprises thecombination of a feature of shape (e.g., a symbol) indicative of whichside of the middle image plane the needle and/or its trajectory is onwhere it traverses the edge of the image area, and a color indicative ofproximity between a middle plane of an image slice and the point atwhich a needle or its trajectory traverses an edge of the image slice.

A pre-determined threshold distance upon which a needle is judged to byin-plane may be, for example, in the range of 0.2 mm to 1 mm. In someembodiments, a pre-determined in-plane threshold distance is 0.5 mm. Insome embodiments, an in-plane threshold distance may be specified by auser. Apparatus may provide a user control by which a user can specifyan in-plane threshold distance.

In some embodiments, an in-plane threshold distance is computed based ona maximum angular separation between an axis of a needle and the planeof an image. In some embodiments, an in-plane threshold distance iscomputed as a function of the distance between opposite edges of theimage and a maximum angular separation between the needle axis and theimage plane. For example, given a distance D between the opposite edgesof the image and a maximum angular separation between the needle and theimage plane of θ, the in-plane threshold distance may be computed asD cos θ  (3)

FIG. 7A shows image 400 bounded by edges 402N, 402E, 402S and 402W.Opposite edges 402E and 402W are spaced apart by a distance D1 in theplane of image 400.f(D1,θ1)=D1 cos θ1  (4)FIG. 7B shows a right triangle 410 that illustrates the computation ofin-plane threshold distance TD1 as a function of D1 and a maximumangular separation θ1, namely the function

It will be appreciated that an in-plane threshold distance may becalculated by functions having similar form, such as, for example,f(D1,θ1)=aD1 cos θ1  (5)where a is a pre-determined constant value.

In some embodiments, an in-plane threshold distance is computed inreal-time as a function of the distance in an image plane between afirst point on a first edge of the image where the needle axis traversesthe first edge and a second point on a second edge of the image areawhere the needle axis traverses the second edge, and a maximum angularseparation between the needle axis and the image plane. For example,given a distance D between a first point on a first edge of the imagewhere the needle axis traverses the first edge and a second point on asecond edge of the image area where the needle axis traverses the secondedge, and a maximum angular separation between the needle axis and theimage plane of θ, the in-plane threshold distance may be computed asD cos θ  (6)

In FIG. 7A, point 406 on edge 402N of image 400 is the point where aneedle axis (not shown) traverses edge 402N, and point 408 on edge 402Wof image 400 is the point where a needle axis (not shown) traverses edge402W. Points 406 and 408 are spaced apart by a distance D2 in the planeof image 400. FIG. 7B shows a right triangle 420 that illustrates thecomputation of in-plane threshold distance TD2 as a function of D2 and amaximum angular separation θ1, namely the functionf(D2,θ2)=D2 cos θ2  (7)

It will be appreciated that an in-plane threshold distance may becalculated by functions having similar form, such as, for example,f(D2,θ2)=aD2 cos θ2  (8)where a is a pre-determined constant value.

In some embodiments, an in-plane threshold distance is computed as afunction of the distance along the needle axis between the edges of theimage and a maximum angular separation between the needle/trajectory andthe image plane. For example, given a maximum angular separation betweenthe needle axis and the image plane of θ and a distance H between thepoints on the needle axis where the axis traverses the edges of theimage, the in-plane threshold distance may be computed as

$\begin{matrix}\frac{H\;\sin\;\theta}{2} & (9)\end{matrix}$

In some embodiments, in-plane threshold distances are computed for edgesseparately. For example, given a maximum angular separation between theneedle and the image plane of θ and a distance D between theintersection of the needle axis and a point on an edge of the imagewhere the needle axis traverses the edge, an in-plane threshold distancefor the edge may be computed asB cos θ  (10)

FIG. 8A shows an ultrasound operating environment 430. An image plane432 is intersected by an axis 434 of a needle at point 437. Axis 434traverses an edge 433 of image plane 432 at point 436 on edge 433. Point436 is spaced apart from point 437 by a distance D in the plane of image432. FIG. 8B shows a right triangle 440 that illustrates the computationof in-plane threshold distance TD as a function of D and a maximumangular separation θ, namely the function.f(D2,θ2)=D2 cos θ2  (11)

It will be appreciated that an in-plane threshold distance may becalculated by functions having similar form, such as, for examplef(D,θ)=aD cos θ  (12)where a is a pre-determined constant value.

Another example of computing in-plane threshold distances for edgesseparately comprises computing an in-plane threshold distance based on amaximum angular separation between the needle axis and the image planeof θ and a distance H between a point on the needle axis where theneedle axis traverses an edges of the image using the functionf(H,θ)=H sin θ  (13)

In FIG. 8A, point 438 is the point on needle axis 434 where axis 434traverses edge 433. Point 438 is separated from point 437 by a distanceH along axis 434. FIG. 8B shows a right triangle 440 that illustratesthe computation of in-plane threshold distance TD as a function of H anda maximum angular separation θ, namely the function.f(H,θ)=H sin θ  (14)

It will be appreciated that an in-plane threshold distance may becalculated by functions having similar form, such as, for examplef(H,θ)=aH sin θ  (15)where a is a pre-determined constant value.

Similar trigonometric relationships for determining an in-planethreshold distance based on the orientation and location of the needleand a maximum angular separation may be used. Maximum angular separationused in determining an in-plane threshold distance may be in the rangeof 0.1 to 1.5 degrees, for example. In some embodiments, the maximumangular separation is 0.5 degrees. In some embodiments, a maximumangular separation may be specified by a user. Apparatus may provide auser control by which a user can specify a maximum angular separation.

Those skilled in the art will appreciate that the dimensions ofultrasound images may depend on the configuration of the ultrasoundtransducer used to acquire the image and the gating of samples (i.e.,the distance of the slice imaged from the transducer). Where this is thecase, methods and apparatus may provide for determination of imagedimensions in real-time.

In some embodiments, a user can mark one or more reference positions onimage 23. A reference position may correspond to a target location atwhich it is desired to place the tip of needle 21, a structure thatcannot be penetrated by needle 21, or the like. Reference positions maybe defined in terms of coordinates of a global coordinate system, suchas, for example, the global coordinate system used by 3D positionsensing system 16 to track position markers 15. Reference positions maycomprise a plurality of points in space.

A reference position marker corresponding to the marked referenceposition may be displayed on image 23. In some embodiments the displayindicates a deviation between the reference position and the location 32at which the further advance of needle 21 would intersect the planerepresented by image 23. An audible signal generated by a suitabletransducer, such as a speaker 35 may also be provided to give the useran indication of the proximity of the tip of needle 21 to the referenceposition at which it is desired to place the tip of the needle and/or anindication of the deviation between the reference position and thelocation 32 at which the further advance of needle 21 would intersectthe plane represented by image 23.

Some embodiments provide a touch screen display and reference positionscan be identified by touching an area of an image displayed on thescreen. In some embodiments, reference positions can be specified bytracing areas on an image using a finger, a stylus or the like. In someembodiments, users may specify an appearance characteristic for a markerthat represents a reference position to differentiate the referencepositions from other reference positions. For example, a user mayspecify that a marker representing a target location at which it isdesired to place the tip of needle 21 be displayed in a first color, andthat a marker representing a structure that cannot be penetrated byneedle 21 be displayed in a second color.

FIG. 9 shows a display 220 according to an example embodiment. Display220 comprises B-mode image 222. Image 222 includes a line 224representing a projection of a needle onto the plane of the bodyrepresented in image 222. Another computer generated line 226 indicatesthe trajectory of the needle. Trajectory-image plane intersection marker228 indicates the location at which the trajectory of the needleintersects the plane of image 222. Reference position marker 230indicates the location of a user-marked reference position. Deviationmarker 232 indicates a deviation between the location of referenceposition marker 230 and trajectory-image plane intersection marker 228.

A reference position marked on an image corresponds to one or morepoints in three dimensional space (e.g., one or more points lying in theplane of the image on which the reference position is marked). Wherefree hand probes, such as probe 12, are used, it is possible that thefield of view and/or image plane corresponding to a current position ofthe probe does not contain a out-of-image plane reference position. Inthese circumstances, users may find it difficult to re-position theprobe so that the reference position is in the plane of the currentimage.

In some embodiments, when the plane of the current image does notcontain a out-of-image plane reference position, a reference positionprojection marker corresponding to a projection onto the current imageplane of the out-of-image plane reference position is displayed. Theprojection of the reference position onto the current image plane maycomprise a projection parallel to the trajectory of a needle. Areference position trajectory-projection marker, which comprises aprojection parallel to the trajectory of a needle, may be used inconjunction with the orientation of the needle as references forre-positioning the probe and/or needle relative to the out-of-imageplane reference position. In some embodiments, a projection of areference position onto a current image plane comprises an orthogonalprojection onto the plane of the current image. A reference positionorthogonal-projection marker, which comprises an orthogonal projectiononto the current image plane, may be used in conjunction with theorientation of the probe as references for re-positioning the probeand/or needle relative to the out-of-image plane reference position.

In some embodiments, when the plane of the current image does notcontain an out-of-image plane reference position, a marker is providedto indicate the proximity of the trajectory of a needle to theout-of-image plane reference position. Such a marker may correspond tothe projection onto the current image plane of the location where theneedle trajectory intersects the plane that contains the referenceposition and is parallel to the current image plane. The projection ofthe trajectory-reference position plane intersection onto the currentimage plane may comprise an orthogonal projection onto the current imageplane (i.e., a projection along projector that is orthogonal to theplane of the current image). A user may position a needle so that itstrajectory intersects an out-of-image plane target by aligning atrajectory-reference position plane orthogonal-projection marker with areference position orthogonal-projection marker.

FIGS. 10 and 11 show, respectively, schematic diagrams 240 and 250 of anultrasound operating environment. FIG. 12 shows an ultrasound imagedisplay 260 corresponding to the ultrasound operating environmentdepicted in diagrams 240 and 250. Display 260 comprises image 262. Indiagrams 240 and 250, a reference position 242, and needles 244, 254 andtheir trajectories 246, 256 are depicted relative to lines 248, 258.Line 248 corresponds to edge 262A of image 262. Line 258 corresponds toedge 262B of image 262. In image 262, lines 264 and 266 represent theorthogonal projections, respectively, of needle 244 and its trajectory246 onto the plane of image 262. Lines 274 and 276 represent theorthogonal projections, respectively, of needle 254 and its trajectory256 onto the plane of image 262.

In diagrams 240 and 250, trajectories 246, 256 intersect lines 248 and258 at trajectory-image plane intersections 246A, 256A. In image 262,trajectory-image plane intersection markers 266A, 276A indicate wheretrajectories 246, 256 intersect the plane of image 262, and correspond,respectively, to trajectory-image plane intersections 246A, 256A.Trajectory-image plane intersection markers 266A, 276A may assist a userin guiding the tips of needles 244 and 254 toward structures depicted inimage 262.

In diagrams 240 and 250, projector 243 extends from reference position242 to lines 248 and 258 to define reference position projection 243A.Projector 243 is perpendicular to lines 248 and 258, and orthogonal tothe plane of image 262. In image 262, reference positionorthogonal-projection marker 263A represents the projection of referenceposition 242 onto image plane 262 according to projector 243, andcorresponds to projection 243A. Reference position orthogonal-projectionmarker 263A may assist a user in positioning a probe to acquire an imagethat contains reference position 242.

In diagrams 240 and 250, lines 249 and 259 are the orthogonalprojections of lines 248 and 258, respectively, onto a plane thatcontains reference position 242 and that is parallel to the plane ofimage 262. Trajectories 246, 256 intersect lines 249 and 259 attrajectory-reference position plane intersections 246B, 256B. Projector247 extends from trajectory-reference position plane intersection 246Bto lines 248 and 258 to define projection 247A. Projector 247 isperpendicular to lines 248 and 258 and orthogonal to the plane of image262. Projector 257 extends from trajectory-reference position planeintersection 256B to lines 248,258 to define projection 257A. Projector257 is perpendicular to lines 248 and 258, and orthogonal to the planeof image 262.

In image 262, trajectory-reference position plane intersectionorthogonal-projection marker 267A represents the orthogonal projectionof trajectory-reference position plane intersection 246B onto imageplane 262 according to projector 247, and corresponds to projection247A. Trajectory-reference position plane intersectionorthogonal-projection marker 277A represents the orthogonal projectionof trajectory-reference position plane intersection 256B onto imageplane 262 according to projector 257, and corresponds to projection257A. Trajectory-reference position plane intersectionorthogonal-projection markers 267A and 277A may assist a user inpositioning a needle so that its trajectory is aligned with referenceposition 242. For example, a user may re-orient a needle so that itscorresponding trajectory-reference position plane intersectionorthogonal-projection marker overlaps the reference positionorthogonal-projection marker in order to align the needle with anout-of-image plane reference position.

In diagrams 240 and 250, projectors 245, 255 extend from referenceposition 242 to lines 248 and 258 to define reference positionprojections 245A, 255A. Projectors 245, 255 are parallel to trajectories246, 256. In image 262, reference position trajectory-projection marker265A represents the projection of reference position 242 onto imageplane 262 according to projector 245, and corresponds to projection265A. Projector marker 265 represents the projection of projector 245onto the plane of image 262. Reference position trajectory-projectionmarker 275A represents the projection of reference position 242 ontoimage plane 262 according to projector 255, and corresponds toprojection 255A. Projector marker 275 represents the projection ofprojector 255 onto the plane of image 262. Reference positiontrajectory-projection markers 265A, 275A and/or projector markers 265,275 may assist a user in positioning a needle so that its trajectory isaligned with reference position 242. For example, a user may interpretthe displacement of a reference position trajectory-projection marker(e.g., marker 265A) relative to a trajectory-image plane intersectionmarker (e.g., marker 266A) as indicative of the displacement of needle244 in a plane parallel to the plane of image 262 that is required toalign the trajectory of the needle with the reference position. Thespacing between projector markers (e.g., markers 265 and 266) may assistusers in identifying the required displacement.

Advantageously, embodiments in which projections of out-of-image-planereference positions are displayed on images may assist users inpositioning an ultrasound probe to acquire an image that contains thereference position and/or positioning a needle for alignment with areference position.

In some embodiments, coded appearance characteristics are applied tomarkers to indicate the distance between an out-of-image plane feature(e.g., reference position, trajectory intersection or the like) and theplane of an image. For example, reference position projection marker263A may be displayed in different colors along a spectrum to indicatethe distance between reference position 242 and its projection 243A ontothe plane of image 262 (e.g., the length of projector 243). In someembodiments, when reference position 242 is on a first side of the planeof image 262, marker 263A is displayed in different colors of a firstspectrum, and when reference position 242 is on a second side of theplane of image 262, marker 263A is displayed in different colors of asecond spectrum, different from the first. For example, as the distancebetween reference position 242 and its projection 243A onto the plane ofimage 262 closes when reference position 242 is on a first side of theplane of image 262, the color in which marker 263A may change along aspectrum running from green to blue, and as the distance betweenreference position 242 and its projection 243A onto the plane of image262 closes when reference position 242 is on a second side of the planeof image 262, the color in which marker 263A may change along a spectrumrunning from red to yellow.

In some embodiments, reference position projection marker 263A may bedisplayed as a marker whose size varies according to the distancebetween reference position 242 and its projection 243A onto the plane ofimage 262 (e.g., the length of projector 243). For example, referenceposition marker 263A may comprise a circle centered at the projection234A of reference position 242 on the plane of image 262 whose diametervaries inversely with the distance between reference position 242 andits projection 243A onto the plane of image 262. Some embodimentscomprise a scale that relates coded appearance characteristic(s) of oneor more markers to the distance between the corresponding out-of-imageplane features and the plane of image 262.

In some embodiments, mutual alignment of out-of-image plane features,in-plane features and/or image planes are signaled by markers havingparticular coded appearance characteristics. Such use of codedappearance characteristics may assist users in obtaining mutualalignment of out-of-image plane features, in-image plane features and/orimage planes. For example, when a reference position lies in an imageplane, a marker indicating the location of the reference position in theimage may appear differently than a marker used to indicate theprojection of the reference position onto the image plane when thereference position does not lie in the image plane. Embodimentsaccording to this example may provide the effect of “lighting-up” thereference position as the image plane is swept through the referenceposition.

In some embodiments, one or more coded appearance characteristics of atrajectory-reference position plane intersection orthogonal-projectionmarker (e.g., marker 267A) and/or a reference positionorthogonal-projection marker (e.g., 263A) is changed to indicate thealignment of the two markers. This feature may serve to alert users tothe fact that the current trajectory of the needle intersects anout-of-image plane reference position.

Other indicators of the distance between the tip of needle 21 and auser-marked target may be provided. For example, FIG. 3 shows a barchart 33 and a read out 34 which both indicate a distance between thetip of needle 21 and the target location in the patient's body. Thedistance may be calculated from the known positions of the tip of theneedle 21 and the target location.

Knowledge of the relative orientations of probe 12 and needle 21 alsopermits the generation of other views which help the user to visualizethe location of needle 21 relative to its intended target. For example,in some embodiments, a display displays both an image taken in thecurrent image plane of transducer 12 and one or more virtual depictionsof a needle intersecting a 3D volume as the needle is tracked. Theillustrated embodiment shows a top view 37A and a side view 37B. In eachcase, the plane of the ultrasound image is represented by a line 38A,the position of needle 21 is indicated by a line 38B, the trajectory ofneedle 21, if advanced in a straight line along its axis, is indicatedby a line 38C. The position at which the trajectory of needle 21 willintersect the plane of the ultrasound images is indicated by a marker38D and the position of the target is indicated by a marker 38E.

In some cases it is desirable to have the needle 21 inserted into thepatient in the same plane of the ultrasound image being taken by probe12. In some such cases, a depiction of the projection of needle 21 onthe surface of transducer array 14 (i.e., the edge of the current imageplane) may be provided to show the relationship of needle 21 to thecurrent image plane. FIGS. 13 and 13A show example projections 167 and167A of this sort. Projections can also show the needle out of thecurrent image plane, as in projection 167B of FIG. 13B.

In some cases, an ultrasound apparatus may be configured to determinehow needle 21 and/or probe 12 ought to be moved in order to bring needle21 into the plane of the ultrasound image. In some embodiments, visibleor audible feedback is provided to indicate when needle 21 is beingmoved farther away from the plane of the ultrasound image 23 or closerto the plane of ultrasound image 23. In some embodiments, the angle ofneedle 21 relative to the plane of ultrasound image 23 is displayed in arepresentation in which shadow is used to indicate the 3D relationshipbetween needle 21 and the plane of image 23 on a 2D image. This isillustrated in FIG. 14, for example. In some embodiments the plane ofthe ultrasound image is controlled automatically or semi-automaticallyto have a desired relationship to the orientation of needle 21. Anexample of such an embodiment is described below with reference to FIG.15.

In some embodiments, biopsy apparatus 19 comprises vacuum assistedbiopsy apparatus, a fire biopsy apparatus or another biopsy apparatusthat can be operated to remove selected tissue from a subject. A vacuumassisted biopsy apparatus may comprise a hollow needle with a lateralopening having a sharp edge. An applied vacuum (suction) draws tissuethrough the opening into the needle bore. When the needle is rotated,the tissue inside the needle bore is cut away by the sharp edge of theopening. The applied vacuum then removes the tissue sample from theneedle. A fire biopsy apparatus operates on a similar principal, exceptthat a cutting member is moved along the needle (fired) to cut away thetissue inside the needle bore. Biopsies taken using such apparatus canobtain several tissue samples from different regions during a singleneedle insertion. In embodiments comprising such biopsy apparatus, anindication of the region that would be sampled (i.e., the region thatwould be cut away) may be provided. This is illustrated in FIGS. 13 and13B, where boxes 160 and 160A indicate on ultrasound images 162 and 162Aregions of tissue that would be cut into aperture 164 of biopsy needle166. FIG. 13A shows a region 165 about aperture 164 that corresponds tobox 160. Determination of the region that would be sampled may comprisedetermining the position and orientation of the lateral opening of thebiopsy needle, determining the region of tissue adjacent to the openingthat would be drawn into the needle by the applied vacuum, determiningthe path that the lateral opening of the biopsy needle would trace if itwere rotated and/or determining the path that the cutting member wouldtravel if it were fired.

In order to indicate the region that would be sampled by a vacuumassisted biopsy, it is desirable to have a description of the needleshaft and features thereon in 3D space. For example, it may be desirableto know the location and orientation of the lateral opening on a vacuumbiopsy needle. In embodiments where needle 21 is connected to handle 19using a coupling which fixes the orientation of needle 21 relative tohandle 19 such that the axis of needle 21 has a predetermined alignmentwith the position markers 15D and 15E, the description of the needleshaft in 3D space can be determined analytically from the location ofthe needle tip, the location of any needle feature relative to the tipand the needle axis, and the predetermined offset of the needle tiprelative to position markers 15D and 15E.

A description of the needle shaft can alternatively be obtained bydetermining the offset of one or more other points along the needleshaft (i.e., points other than the needle tip, such as the ends of alateral opening of a vacuum biopsy needle) relative to positionmarker(s) on the biopsy apparatus. To determine the offset of such aposition, the needle may be placed in a slide track or any othersuitable device for constraining the movement of the needle to its axis,and the transmit coordinate of the needle tip is determined from itspre-determined offset from position markers 15D and 15E. Then the needleis then moved a distance along the direction of its axis. In someembodiments, this distance is greater than 5 mm.

Where the needle is advanced in the direction pointed out by the needletip, the point in space where the needle tip was before the movementwill be occupied by a point on the needle shaft that is the distance ofthe movement from the needle tip. Where the needle has been retracted inthe direction opposite the direction pointed out by the needle tip, theneedle tip will occupy a point in space where a point on the needleshaft that is the distance of the movement from the needle tip wasbefore the movement. In either movement case, the position of the pointalong the needle shaft at the movement distance from the needle relativeto position markers 15D and 15E can be determined by calculating theposition of the point along the needle shaft relative to the needle tipfrom the difference in the needle tip position before and after themovement, and then relating the position of the point along the needleshaft relative to the needle tip to the global coordinate system usingthe location of the needle-tip in the global coordinate system.

It has been found that when positions are determined using some magneticposition sensing systems, excessive separation between position markers15 and position base unit 17 and/or the presence of metals near positionmarkers 15 and/or position base unit 17 can reduce the quality of thetracking signals that pass between markers 15 and position base unit 17.As a result, the accuracy of position and orientation informationobtained by position sensing system 16 may be diminished. Often accuracycan be restored by adjusting the position markers (i.e. moving and/orrotating the needle and/or probe) or the position base unit and/or bymoving the metal away from the markers and/or position base unit. Toalert users to the condition of possibly diminished accuracy of positionand orientation information, the quality of the tracking signal may bemonitored and an indicator thereof provided. Graphical indicator 168 inFIG. 13 and graphical indicator 168A in FIG. 13A are examples of suchindicators. It will be appreciated that tracking signal quality may berepresented in many different graphical forms or representednumerically. The quality of the tracking signals may be inferred orestimated from, for example, the mean received power of the trackingsignals, the error rate in a known training sequence, or the like.

In some embodiments, some or all of the display (e.g., line 24representing a projection of the needle 21 and line 30 representing thepath that will be followed by the tip of needle 21) may be hidden ordisplayed in a different manner (e.g. dimmed, flashed or the like) whenthe quality of the tracking signal falls below a threshold so that theuser is not provided without warning with possibly invalid positioninformation. In some embodiments, an audible and/or visual alarm istriggered when the tracking signal quality falls below a threshold.

In some embodiments, the fact that the position and orientation ofultrasound probe 12 may be measured in real time is used to build a 3Dmodel of a patient's tissues. This is illustrated in FIG. 16 which showsa flow diagram of a method 40 for generating a 3D model. In method 40,an ultrasound frame is received at block 41 and positions of thetransducer position markers 15 are detected at block 42. Blocks 41 and42 are performed essentially simultaneously. The position andorientation of the transducer is determined in block 43 and atransformation between the coordinates of pixels in ultrasound frame 41and the 3D coordinates of voxels in a 3D model being built is determinedat block 44. In block 45, the transformation determined in block 44 isused to transform the pixels of the 2D ultrasound frame received atblock 41 into corresponding voxels. In block 46, the transformed data isadded to a 3D model 47. It can be appreciated that if multiple frames ofan ultrasound image are acquired over time then, as long as the positionof the patient P does not change significantly relative to 3D positionbase unit 16, a 3D model of the patient can be readily obtained.

The 3D model may comprise a 3D array of voxels. Each voxel may contain avalue indicating the degree to which ultrasound signals echo from acorresponding location within the patient.

Note that 3D model 47 may continue to be added to while ultrasoundframes are received for other purposes. For example, a user may bemoving ultrasound transducer 12 over a patient P in order to find alocation of interest. While the user is doing so, a system may bebuilding or adding to a 3D model 47 of the portions of the patient thatare visualized by transducer 12.

Once a 3D model has been obtained, then the 3D model may be exploited ina wide variety of ways, some of which are described below. For example,the 3D model may be used to generate 2D images on any plane crossingthrough the 3D model. A representation of needle 21 may also be shown insuch images. In some embodiments user controls allow users to select thepoints of view of the images and the types of images being displayed.Once a 3D model has been acquired, an image showing the approach ofneedle 21 to a structure of interest may be displayed regardless of thecurrent orientation of transducer 12. Indeed, as long as the patient Premains in a stable position relative to 3D position base unit 17,transducer 12 is not even required to guide the introduction of needle21 to a desired location inside the patient P.

FIG. 17 shows an example system 50 which has this functionality. Insystem 50, position markers 15F and 15G are attached to patient P. Forexample, position markers 15F and 15G may be stuck to a patient's skinusing adhesive. The position markers may be stuck at locations that tendto move with the patient, for example, to the skin overlying a patient'svertebrae, hips or collar bone.

The positions of position markers 15F and 15G are monitored by positionsensing system 16. Biopsy assembly 19 is also monitored by positionsensing system 16. Position data from position sensing system 16 isprovided to the ultrasound system which displays an image taken from the3D model. The image may be a 2D image, such as a cross-section derivedfrom the 3D model or a 2D rendering derived from the 3D model. In someembodiments, the display 52 is a 3D display such that a user can see a3D image.

A line 53 representing needle 21 may be shown in the image. If patient Pmoves slightly then the motion is detected by position sensing system 16which continually monitors the positions of position markers 15F and 15G(there may be additional position markers on the patient P). In general,one or more position markers may be on patient P in an embodiment likeembodiment 50. This information is used to update the location of needle21 relative to anatomical structures displayed in an image 54 on display52.

FIG. 18 is a block diagram of apparatus 60 according to an embodiment.Apparatus 60 includes a biopsy assembly 19, an ultrasound probe 12, and3D position sensing system 16, as discussed above. Ultrasound probe 12is connected to an ultrasound unit 62 which controls ultrasound probe 12to generate appropriate ultrasound signals and which processes receivedultrasound echos to yield ultrasound scan data.

In the illustrated embodiment, the ultrasound scan data comprises aseries of 2D ultrasound scans. For example, 2D ultrasound scans may beobtained at a rate of several scans per second. 3D position sensingsystem 16 generates spatial orientation data for the ultrasound probe 12that corresponds to each of the ultrasound scans. This spatialorientation data can be used to transform a local coordinate system ofeach one of the scans to a coordinate system used in a 3D model of apatient.

A scan combiner combines scans 63 using spatial orientation data 64 intoa 3D model 66. A 3D image rendering system 67 displays a rendering of 3Dmodel 66 on a display 68. System 60 also obtains spatial orientationdata 69 for biopsy needle 21. A needle tip position calculation 70provides a position for the tip of needle 21 in the coordinate space of3D model 66. The needle tip position is provided to 3D image renderingsystem 67. A needle alignment calculation 72 establishes a lineextending along the axis of needle 72. This line intersects anatomicalstructures that needle 21 will encounter if it is advancedlongitudinally into the patient. The line generated by needle alignmentsystem 72 is also provided to 3D image rendering system 67.

System 60 also includes a 2D image rendering system which can displayultrasound scans 63 and which can super-pose on those ultrasound scansimages representing the position of the tip of needle 21 as well as theline which provides information about alignment of needle 21. 2D imagesrendered by 2D image rendering system 74 may be displayed on display 68.Display 68 may comprise a computer monitor, for example.

FIG. 19 shows an example image that may be displayed on a display 68.The image shows anatomical structures, including an anatomical structureof interest. Also shown are a current position of needle 21, indicatedby line 77, an extension 77A of line 77 which indicates the trajectorythat would be followed by needle 21 and a point 77B indicating the endof needle 21. As described in relation to FIGS. 1 and 3, variousgraphical and audible indications may be given to alert the user to thecurrent relationship between the tip of needle 21 and the targetposition within the patient.

In the illustrated embodiment, a zoom control 79, a multi-axis rotationcontrol 80 and pan-control 82 are provided to allow the user to zoom,rotate and pan the image 76 generated from the 3D model. These controlsmay be activated by way of a graphical user interface or by way of otheruser interface elements. In some embodiments, a user can manipulateimage 76 using a pointing device such as a mouse, track ball, track pad,touch screen or the like to pan, rotate and zoom image 76 in a desiredmanner. In some embodiments, zoom and rotation of image 76 areautomatically adjusted so that, as the user manages to get the tip ofneedle 21, as indicated by line 77 and point 77B, close to the targetthe image automatically zooms in and rotates to an angle that best showsthe approach of needle 21 to the target.

In some locations, a vacuum assisted biopsy needle is provided. In suchembodiments, the biopsy apparatus 19 may comprise a control, such as apush button, foot pedal, or the like, which enables a user to triggerthe biopsy needle to acquire a biopsy sample. In some such embodiments,the control is interfaced to the ultrasound system in such a manner thatthe current position of needle 21 is preserved upon actuation of thecontrol. A real-time ultrasound image, a 3D model or an image extractedfrom a 3D model may also be preserved, if available. This permits laterexamination of the exact position of the needle at the time each samplewas obtained. These images and/or position information may be archivedfor future reference.

In some embodiments, the ultrasound apparatus is configured toautomatically determine whether a 3D model of the area of the patient iscomplete. FIG. 20 shows a method 90 according to one example embodiment.In block 91, scan data is acquired in block 91A and the correspondingprobe position and orientation is acquired in block 91B. In block 92 thedata is mapped to voxels in a 3D model. In block 93, it is determinedwhether the 3D model is complete. For example, block 93 may compriseconsidering whether a threshold number or density of voxels have not yetbeen supplied with image data. If the 3D image is not yet complete thenmethod 90 loops to block 91 on path 94 to acquire more data. Otherwise,method 90 continues to block 95. In block 95 needle 21 is calibrated,for example as described above. Note that block 95 may be performedbefore or after acquiring the 3D model. In block 96 the position andorientation of needle 95 are determined. Method 90 may then continue toallow a user to visualize the position of needle 21 while taking abiopsy or doing other medical procedures, as described herein forexample.

In some embodiments, an ultrasound system is configured to provideinstructions that will be useful in helping a user to obtain suitablealignment between a needle 21 and a target location. FIG. 21 shows anexample system 100 in which a position base unit 102 determinespositions of a ultrasound transducer and a needle. A block 104 computesan image plane of the transducer from the transducer position data. Ablock 106 computes a line along which needle 21 is aligned (e.g., atrajectory of needle 21) from the needle position data. Block 108computes an angle between the needle and the image plane of thetransducer.

In some embodiments, block 108 determines one or more of the following:

-   -   coordinates of a point at which the line determined in block 106        intersects the image plane of the transducer;    -   whether the point at which the line determined in block 106        intersects the image plane of the transducer lies in the        field-of-view of the image;    -   a distance along the line determined in block 106 between the        image plane of the transducer and the tip of needle 21;    -   a distance along a line normal to the image plane of the        transducer between the image plane and the tip of needle 21;    -   which side of the image plane of the transducer the tip of        needle 21 is on; and    -   the like.

An instruction generator 110 generates audio and/or visual instructionsthat assist the user to obtain a desired alignment between the needleand the image plane of the transducer. Instructions may be provided byan audio output system 112 and/or a display 114. In some embodiments,block 110 determines one or more locations and/or coded appearancecharacteristics for a marker, line or other display feature shown ondisplay 114 that corresponds to information determined in block 108.Block 110 may determine such locations and/or coded appearancecharacteristics by way of a function, a look-up table, or the like. Insome embodiments, instruction generator 100 generates a scale fordisplay on display 114 that relates coded appearance characteristics toinformation determined in block 108.

FIG. 14 shows an example display 11 in which a representation of theneedle and the image plane of a transducer are both displayed with anangle indicated. Audio instructions and/or text instructions and/orvisual instructions may be provided to assist the user in manipulatingthe needle to provide the proper needle orientation.

Application of the invention is not limited to taking biopsy samples.For example, the systems described herein may be applied to find acorrect location within the body for the introduction of a drug, such asa anesthetic, or a radioactive seed for cancer treatment or the like.For example, the system may be used to position a catheter to introducean epidermal anesthetic.

In some embodiments, one or more position markers 15 are built intoultrasound probe 12 and/or needle assembly 19. In other embodiments,position markers may be mounted to probe 12 and/or needle assembly 19using a clip or other removable fastener. In some embodiments, needle 21is large enough in diameter that a position marker 15 can be provided inneedle 21 or even at or near the tip of needle 21. Such embodiments canprovide better information regarding the location of the tip of needle21 in cases where it is possible that needle 21 may bend slightly inuse.

FIG. 22 shows an example ultrasound probe 130 having built-in positionmarkers 15. A cable assembly 132 servicing probe 130 includes signalconductors 21A, 21B and 21C that carry signals from position markers 15to position base unit 17.

In some applications, it is desirable that the probes can beinterchanged without having to perform calibration or load newcalibration parameter information. This can be facilitated during probesensor assembly by ensuring that the position/orientation markers thatprovide location and orientation information for the probes areidentically situated with respect to the transducer array of each of theprobes. In particular, the position/orientation markers may be placed atthe same location in different probes and at the same rotation angle. Insome embodiments, one or more axes of the coordinate system according towhich the position and orientation of the marker is determined may bealigned with corresponding axes of the ultrasound transducer (e.g., theaxes defining the plane of the ultrasound image).

FIG. 23 shows a side elevation view of example embodiment of a markerpositioning apparatus 170 that may be used for precisely mounting aposition marker 182 into a bore 180 of a probe 171. FIG. 24 shows a topplan view of positioning apparatus 170. Probe 171 and a position baseunit 172 are mounted on a jig 174. In some embodiments, the distancebetween the probe 171 and position base unit 172 is in the range of 8and 40 cm. In order that position markers can be positioned uniformlywith respect to the transducer arrays of different probes, mounting aprobe in jig 174 should result in substantially the same spatialrelationship between the transducer array of the probe and position baseunit 172. In the illustrated embodiment, jig 174 comprises a seat 175that conforms to a portion of the exterior of probe 171. Seat 175ensures that all probes of the same model as probe 171 will beidentically mounted in jig 174. As a result, the position andorientation of the transducer arrays of such probes with respect toposition base unit 172 will be the same as the position and orientationof transducer array 184.

In some embodiments, jig 174 comprises a plurality of seats, eachconfigured to conform to a portion of a different probe model andpositioned on jig 174 such that the transducer arrays inside thedifferent model probes are substantially identically situated withrespect to position base unit 172. In some such embodiments, the seatsmay be removable and interchangeable. It will be appreciated that othermeans could be employed to facilitate uniform mounting of probes in jig174. For example, conductive plates could be provided on jig 174 andprobe 171 in locations such that when probe 171 is properly mounted onjig 174 an electric current flows across the plates to trigger a signalindicative of proper mounting.

With probe 171 and position base unit 172 mounted on jig 174, a positionmarker 182 is inserted into bore 180. A real-time display of theposition and orientation of marker 182 is provided to a user. FIG. 25 isa screen-capture of an example real-time display 200 of the position 202and orientation 204 of a marker according to an example embodiment. Theuser can use this information to guide position marker 182 into adesired relationship with position base unit 172. Typically, a user willwant to guide marker 182 into a pre-determined position (e.g., X, Y, Zcoordinate) and orientation (e.g., azimuth, elevation and rotationangles) with respect to position base unit 172, so that markers for anumber of different probes can be guided into the same pre-determinedposition and orientation.

Bore 180 and jig 174 may be configured to facilitate guiding positionmarker 182 into a desired position and orientation. In such embodimentsthe body of position marker 182 may be shaped to have at least onefeature that corresponds to a coordinate axis by which its position ismeasured. For example, FIG. 26 shows a cylindrically shaped positionmarker 190. The axis 191 of marker 190 corresponds to one axis 191X ofthe three axes (191X, 191Y and 191Z) by which the position of marker 190is measured. Bore 180 can be defined in probe 171 so that a feature ofmarker 182 that corresponds to a coordinate axis by which its positionis measured is aligned (or at least substantially aligned) with an axisof position base unit 172 when marker 182 is inserted in bore 180. Inthe embodiment illustrated in FIG. 23, the axis of bore 180 is alignedwith an axis of position base unit 172 and position marker 182 isgenerally cylindrical with the axis of its body corresponding to an axisby which position base unit 172 defines the position of marker 182.Inserting marker 182 along bore 180 causes the axis of the body ofmarker 182 to be aligned with an axis by which position base unit 172defines the position of marker 182. Provided that marker 182 fits snuglyinside bore 180, the axial alignment will substantially determine theazimuth and elevation angles of marker 182 and the position of marker182 along the other axes by which position base unit 172 defines theposition of marker 182. Thus a user aligning marker 182 to apre-determined orientation and position need only adjust marker 182translationally along bore 180 and rotationally about its axis.

In the illustrated embodiment, jig 174 and bore 180 are also configuredso that end 180A of bore 180 is a pre-determined distance fromtransducer array 184. This allows the position of position marker 182relative to the position base unit 172 to be fixed precisely at a knowndistance from position base unit 172 by abutting marker 182 against end180A. In some embodiments, a spacer may be deposited at end 180A of bore180 in order to increase the minimum separation of position marker 182from position base unit 172. In some such embodiments, the depth of bore180 is ascertained and a spacer of a pre-determined dimension isinserted to make bore 180 have a known effective depth.

In some embodiments, bore 180 is slightly larger than the cross-sectionof marker 182. This permits marker 182 to be adjusted within bore 180 tomore precisely locate marker 182 with respect to position base unit 172.The ability to adjust marker 182 in bore 180 is desirable where thespatial relationship between bore 180 and transducer array 184 maydiffer between probes, such as where the bore is formed in anoff-the-shelf probe. Where bore 180 is slightly larger than thecross-section of marker 182, a liquid or gel adhesive can be applied toeither or both of bore 180 and marker 182, and marker 182 held in bore180 at the desired position and orientation with respect to positionbase unit 172 until the adhesive has set.

Because the quality of the tracking signal is related to the accuracy ofthe position and orientation information obtained therefrom, it ispreferable that the tracking signal be of at least a threshold qualitywhen positioning marker 182 in bore 180. In some embodiments a very highquality value may indicate either high magnetic field distortion ormalfunctioning of position marker 182. In some such embodiments, it ispreferable that the quality value be less than a threshold value duringorientation of position marker 182. Some embodiments provide users withan indication of tracking signal quality. Display 200 in FIG. 25includes a numerical indication of tracking signal quality 206. Otherembodiments may provide graphical or audible indications of trackingsignal quality.

In some embodiments, the plane of an ultrasound image obtained by probe12 can be steered electronically. In some such embodiments, an operatingmode is provided in which the plane of the ultrasound image isautomatically steered so that the needle lies perpendicular to the planeof the ultrasound image. This can enhance visibility of the needle. Inother embodiments the plane of the 2D ultrasound image is steeredautomatically to be in the same plane as the needle. Steering may beaccomplished, for example, by selecting a group of transducers from atransducer array in probe 12 and/or performing beamforming ontransmitted and received ultrasound signals.

FIG. 15 shows schematically ultrasound apparatus 140 which includes abeam-steering capability. In the illustrated embodiment, an angle θbetween the plane of the ultrasound image and a reference plane 141 oftransducer 142 can be electronically varied. In some embodiments, θ iscontrolled to make the plane of the ultrasound image perpendicular tothe axis of needle 21.

In some cases, 3D data may also be available from another source. Forexample, there may be magnetic resonance imaging (MRI) or computedtomography (CT) results for the patient that provide 3D volume data. Insome embodiments, the 3D model derived from the ultrasound data isco-registered to the other 3D image data. This permits variousalternatives. For example, the position of the needle may be shownrelative to structures shown in an MRI or CT image. In some embodiments,a composite image is created from one or more of the ultrasound images,the magnetic resonance image and the CT image. The composite image maycomprise high contrast boundaries shown in any one or more of theseimaging modalities. The position of needle 21 may then be shown incomparison to the composite image.

FIG. 14 shows an example shadow representation which could be displayedon a 2D monitor to indicate the 3D relationship of a needle to ananatomical structure.

In some embodiments, the origin of the coordinate system for the 3Dmodel is set automatically. There are various ways to achieve this. Forexample, a user may move probe 12 over a patient P to acquire ultrasounddata. In the course of doing so, a number of ultrasound frames areacquired. The ultrasound frames may provide reasonably complete coverageof a particular region in space. The ultrasound unit may be configuredto identify a region of space that is reasonably completely covered bythe acquired ultrasound data (for example, a region in which a densityof voxels for which there are one or more corresponding pixels in theultrasound data) exceeds some threshold density. The system may thenestablish an origin for the 3D model with reference to the regionidentified. The origin may be determined for example with reference to acentroid of the identified volume.

In another embodiment, a coordinate system for the 3D model isestablished with reference to specific positions for probe 12. In anexample embodiment illustrated in FIG. 27, a user places probe 12 at afirst location 150A on a first side of a region of patient P that is ofinterest and indicates by triggering a control when the transducer is inthe desired position. The user may then move probe 12 to a secondlocation 150B on another side of the region of interest and trigger thesame or different control to indicate that the transducer is now locatedat the other side of the region of interest. The origin of thecoordinate system for the 3D model may then be set with reference to thetwo positions. For example, the origin may be set at one of thepositions. The orientation of an axis, such as the X-axis in FIG. 27 maybe set with reference to the difference vector between the twopositions. The orientation of another coordinate, such as the Z-axisshown in FIG. 27 may be determined with reference to the orientation ofprobe 12 in one or both of the positions. Once the X and Z axises aredetermined, the Y-axis is determined automatically.

The 3D model is not limited to having a Cartesian coordinate system. The3D model may also be defined using some other coordinate system such asa spherical coordinate system or a cylindrical coordinate system.

It is not mandatory that the 3D model be built in real time. In someembodiments, the 3D model is built in real time as new ultrasound framesare received. In other embodiments, a plurality of ultrasound frames arereceived and saved and the 3D model is then built from the received andsaved ultrasound frames. Position/orientation information correspondingto each ultrasound frame may be saved. In either case, where aparticular voxel of the 3D model corresponds to pixels in multipledifferent ultrasound frames, then the value for the voxel may be set ina way that selects one of the ultrasound frames or, in the alternative,combines the values for the corresponding pixels in the differentultrasound frames in some manner (such as taking an average or weightedaverage or median of values corresponding to the voxel from two or moreultrasound frames).

Some example ways in which a single one of the ultrasound frames may beselected to provide the value for a voxel are:

-   -   the voxel value may be set to a value corresponding to a        most-recent ultrasound frame having a pixel corresponding to the        voxel;    -   the voxel may be set to a value determined from a best-quality        one of a plurality of ultrasound frames according to a suitable        image quality measure;    -   etc.

Some example ways in which pixel values from multiple ultrasound framesmay be combined to yield a voxel value include:

-   -   averaging the pixel values;    -   performing a weighted average of the pixel values;    -   where there are three or more corresponding pixel values,        rejecting any outlying pixel values and combining the rest in a        suitable manner (such as by averaging or weighted averaging);    -   etc.

FIG. 28 illustrates a possible relationship between the coordinates of apixel in a single ultrasound frame and the coordinate system for a 3Dmodel. In this example, the coordinate system for each ultrasound frameis a polar coordinate system having an origin located at a point 15A atcoordinates X₀, Y₀, Z₀ in the coordinate space of the 3D model. Thelocation of point 15A is determined by the position of probe 12 when theultrasound frame is acquired. This position is indicated by vector K4.The plane of the ultrasound image is oriented such that the normal tothe plane is defined in this example by the vector K7. The direction ofthe ultrasound image is defined by the vector O1. Given thisrelationship then a pixel identified by the coordinates R, θ in thecoordinate system of the ultrasound frame corresponds to the voxel at alocation X′, Y′, Z′ in the coordinate space of the 3D model. Conversionbetween coordinates of the ultrasonic image and voxels of the 3D modelmay be implemented as matrix multiplications, for example.

Some anatomical structures, such as breasts, may move around duringimage acquisition. In such cases, a 3D image may not be true to thecurrent position of the anatomical structure. In such cases, the probe12 itself may be used as reference location. The user may be instructedto keep the probe in one place on the breast or other moveable tissue.

In an example embodiment illustrated in FIG. 29, imaging of movabletissue, such as a breast, is performed while the breast or other tissueis held in a rigid or semi-rigid form. For breast imaging the form maybe conical, for example. In the embodiment illustrated in FIG. 29, form120 is a conical form. One or more position markers 15 is provided onform 120. Form 120 is made of a material, such as a suitable plastic,that is essentially acoustically transparent at ultrasound frequenciesso that ultrasound images may be acquired through form 120. Apertures122 may be provided in form 120 to permit introduction of a needle suchas a needle 21 of biopsy apparatus 19. In some embodiments form 120 ismesh-like or penetrated by a regular array of apertures. In thealternative, a needle 21 may simply pierce form 120 at a desiredlocation. Form 120 may be a disposable item.

In an example application, a woman's breast is inserted into form 120such that the breast tissue fills and conforms substantially to theshape of form 120. A 3D model of the breast tissue is acquired byrunning an ultrasound probe 12 over the outer surface of form 120. Thevoxels of the 3D model are referenced to the locations of the positionmarkers 15 on form 120. After the 3D model has been acquired, form 120prevents the breast tissue from shifting relative to the positionmarker(s).

A practitioner can then introduce needle 21 of biopsy apparatus 19 intoa desired location in the woman's breast while following progress of theneedle 21 on a display. Optionally, ultrasound images may be taken asneedle 21 is introduced to provide real-time ultrasound images of thetissues through which needle 21 is being introduced.

In another example application, an ultrasound image of a portion of apatient's spinal column is obtained and the techniques described hereinare applied to follow a catheter as its tip is inserted into thepatient's epidural space. FIG. 30 illustrates such an application. Oneor more position markers 15 may be adhered to the patient during thisprocedure. For example, the position markers may be adhesively butremovably affixed to the patient's skin. Positions of the one or moreposition markers on the patient may be monitored and used to maintaincorrespondence between sensed positions of a needle and a 3D model.

FIG. 31 shows an ultrasound system 310 according to an exampleembodiment. System 310 comprises a controller 311 connected to anultrasound probe 312, a display 313, a user input device 314, a 3Dposition sensing system 316, a needle 317, and a position base unit 305.

Ultrasound probe 312 emits ultrasound pulses into the body of patient P.Ultrasound pulses emitted by probe 312 are reflected off of structuresin the body of patient P. Probe 312 receives reflected ultrasound pulsesthat return in its direction. Controller 311 may be configured tocontrol aspects of the operation of ultrasound probe 312. For example,controller 311 may control the transmission of pulses from ultrasoundprobe 312 and/or the gating of samples of reflected pulses received atultrasound probe 312.

Controller 311 may comprise one or more central processing units (CPUs),one or more microprocessors, one or more field programmable gate arrays(FPGAs), application specific integrated circuits, logic circuits, orany combination thereof, or any other suitable processing unit(s)comprising hardware and/or software capable of functioning as describedherein.

Controller 311 comprises a memory 315. In some embodiments, memory 315is external to controller 311. Controller 311 may be configured to storedata representative of signals acquired by probe 312 in memory 315.Controller 311 processes ultrasound data acquired from ultrasoundtransducer 312. In some embodiments, controller 311 is configured togenerate ultrasound image data from ultrasound data acquired by probe312. For example, memory 315 may comprise instructions that whenexecuted by controller 311 or when used to configure controller 311cause controller 311 to generate a B-mode image from ultrasound dataacquired by probe 312. Controller 311 may comprise an analog or digitalbeamformer for use in processing echo signals to yield image data.Controller 311 may be configured to store image data that it generatesin memory 315.

Either or both of controller 311 and probe 312 may optionally be part ofan ultrasound machine that is commercially available. Controller 311 andprobe 312 may be of any known or future developed type.

3D position sensing system 316 includes one or more position markers(not shown) on each of probe 312 and needle 317. The position markers onprobe 312 and needle 317 communicate with position base unit 305. 3Dposition base unit 305 measures the locations of the position markersrelative to a global coordinate system. Where three position markers arelocated on a rigid body and not located along a common line, theorientation of the rigid body is uniquely determined by the positions ofthe three position markers. In some embodiments, probe 312 and needle317 comprise rigid bodies having at least three position markers thatare not located along a common line.

In some embodiments, 3D position sensing system 316 comprises 3Dposition sensing technology that permits both the location andorientation of a single position marker to be determined by 3D positionbase unit 305. In some such embodiments, the location and orientation ofprobe 312 and/or needle 317 may be determined from information providedby as few as one position marker.

3D position base unit 305 is connected to controller 311. In someembodiments, 3D position base unit provides location and/or orientationinformation for markers on probe 312 and/or needle 317 to controller311. In some embodiments, position base unit 305 determines a spatialdescription of probe 312, needle 317 and/or features thereof (e.g., tipof needle 317, the plane of an ultrasound image derived from ultrasounddata acquired by probe 312, etc.) based on information provided by theposition markers, and provides such description(s) to controller 311. Aspatial description may comprise information specifying location and/ororientation in space. The spatial description of probe 312, needle 317and/or features thereof may be specified in terms of any suitable globalcoordinate system (e.g., Cartesian, spherical, cylindrical, conical,etc.).

Controller 311 is configured to generate images from ultrasound data anddisplay such images on display 313. In some embodiments, controller 311is be configured to generate and display on display 313 imagescomprising graphical overlay elements that represent features of anultrasound operating environment relative to the plane of an ultrasoundimage. Such graphical overlay elements may comprise, for example, lines,markers and the like of the type shown herein in images/views/displays23, 37A, 37B, 68, 162, 162A, 220, 260.

Input device 314 provides user input to controller 311. In theillustrated embodiment, input device 314 comprises keyboard 314A andcomputer mouse 314B. Input device 314 may comprise other userinterfaces. In some embodiments, display 313 comprises a touch screen,which may form part of input device 314.

A user may use input device 314 to control aspects of the operation ofcontroller 311. Input device 314 may provide controls for manipulatingimages generated by controller 311. For example, a user may interactwith input device 314 to control the gating of samples received atultrasound probe 312 and thereby change the image displayed bycontroller 311 on display 313. Control may be provided for other aspectsof controller 311, such the selection, determination and/or appearanceof graphical elements overlaid on ultrasound images (e.g., lines andmarkers of images/views/displays 23, 37A, 37B, 68, 162, 162A, 220, 260).

In some embodiments, input device 314 is operable to indicate areference position on an image displayed on display 313. In some suchembodiments, controller 314 is configured to register a location of areference position indicated by a user on an image displayed on display313. The location of the reference position may comprise locations ofone or more pixel in an image display on display 313 and/or one or morecoordinates in a global coordinate system.

In some embodiments, controller 311 may be configured to determinespatial descriptions of features of an ultrasound operating environmentbased on information provided by position base unit 305 (e.g., locationand/or orientation information for position markers on probe 312 and/orneedle 317, spatial descriptions of probe 312 and/or needle 317, or thelike). For example, controller 311 may be configured to determine one ormore of:

-   -   a location of probe 312;    -   a location of needle 317;    -   a location of an image acquired by probe 312;    -   a plane of a location of an image acquired by probe 312;    -   a trajectory of needle 317;    -   a location of the longitudinal axis of needle 317;    -   a location of an intersection of needle 317 with the plane of an        image acquired by probe 312;    -   a location of an intersection of the longitudinal axis of needle        317 with the plane of an image acquired by probe 312;    -   an angle between the longitudinal axis of needle 317 and the        plane of an image acquired by probe 312;    -   a location of a reference position indicated on an image;    -   a plane containing a reference position and parallel to a plane        of an image;    -   a location of an intersection of the longitudinal axis of needle        317 with an arbitrary plane, such as, for example, a plane        containing a reference position indicated on an image and        parallel to a plane of another image;    -   a projector that is orthogonal to a plane of an image acquired        by probe 312 and that extends from a feature of an ultrasound        environment to the plane of the image;    -   a projector that is parallel to a trajectory of needle 317 and        that extends from a feature of an ultrasound environment to the        plane of an image acquired by probe 312;    -   a distance between a reference position and an image acquired by        probe 312 along a projector from the reference position to the        image that is orthogonal to the plane of the image;    -   a distance between a reference position and an image acquired by        probe 312 along a projector from the reference position to the        image that is parallel to the longitudinal axis of needle 317;    -   a distance between the location of an intersection of the        longitudinal axis of needle 317 with an arbitrary plane and an        image acquired by probe 312 along a projector from the        intersection to the image that is orthogonal to the plane of the        image;    -   a distance between the location of an intersection of the        longitudinal axis of needle 317 with an arbitrary plane and an        image acquired by probe 312 along a projector from the        intersection to the image that is parallel to the longitudinal        axis of the instrument.        Controller 311 may determine such spatial descriptions using any        suitable combination of mathematical operations and/or        techniques, such as mapping (e.g., between image pixels and        points in a global coordinate system), interpolation,        extrapolation, projection or the like. Spatial descriptions of        features of an ultrasound operating environment may comprise        locations of pixels in an image, and/or coordinates in a global        coordinate system.

In some embodiments, controller 311 is configured to determineprojections of features of an ultrasound operating environment ontoplanes of ultrasound images. Controller 311 may be configured todetermine projections along various projectors, such as, for example,projectors orthogonal to a plane of an ultrasound image acquired byprobe 312, projectors parallel to a trajectory of needle 317, or thelike. For example, controller 311 may be configured to determine one ormore of:

-   -   an orthogonal projection of at least a portion of the        longitudinal axis of needle 317 onto a plane of an ultrasound        image;    -   an orthogonal projection of the tip of needle 317 onto a plane        of an ultrasound image;    -   an orthogonal projection of a reference position onto a plane of        an ultrasound image;    -   an orthogonal projection of an intersection between the        longitudinal axis of needle 317 and an arbitrary plane, such as,        for example, a plane containing a reference position indicated        on an image and parallel to a plane of another image, onto a        plane of an ultrasound image;    -   a projection of a reference position onto a plane of an        ultrasound image according to a projector that is parallel to        the longitudinal axis of needle 317; and    -   an orthogonal projection of a projector onto a plane of an        ultrasound image, such as, for example, an orthogonal projection        of a projector that is parallel to the longitudinal axis of        needle 317 and that extends from a feature of an ultrasound        environment to a plane of an ultrasound image.

In some embodiments, controller 311 is configured to indicate on display313 features of an ultrasound operating environment and/or projectionsof such features onto a plane of an ultrasound image. For example,controller 311 may be configured to indicate such features and/orprojections on display 313 by causing graphical elements to be displayedon display 313. Such graphical elements may be displayed as overlays onultrasound images displayed on display 313. In some embodiments,controller 311 is configured to determine an image location for displayof a graphical overlay element corresponding to such a feature orprojection by mapping a spatial description of the feature or projectionto pixel(s) of an ultrasound image. For example, controller 311 may beconfigured to graphically overlay a marker representing an orthogonalprojection of a reference position onto a plane of an ultrasound imageat an image location corresponding to the spatial description of theprojection.

In some embodiments, controller 311 is configured to determine if all orpart of a feature of an ultrasound operating environment, or aprojection thereof onto a plane of an ultrasound image, lies in thefield-of-view of an ultrasound image. For example, controller 311 may beconfigured to determine if an intersection of a trajectory of needle 317with a plane of an ultrasound image shown on display 313 lies in thefield-of-view of the ultrasound image.

In some embodiments, controller 311 is configured to determinedisplacements between features of an ultrasound operating environment.For example, controller 311 may be configured to determine thedisplacement between two planes, between a plane and a point in space,between two points in space, and the like. Controller 311 may beconfigured to display graphical and/or numerical indications of suchdisplacements on display 313.

In some embodiments, controller 311 is configured to determine one ormore coded appearance characteristics of markers or lines to conveyspatial relationship information pertaining to features of an ultrasoundoperating environment. Coded appearance characteristics may comprise,for example, size, size, color, intensity, shape, linestyle, or the likethat vary according to the information the characteristics are meant toconvey. Coded appearance characteristics may vary continuously (e.g.,color, brightness or the like along a spectrum; e.g., size, thickness,length, etc.) to convey continuous information (e.g., distance, angle,etc.) or may vary discretely (e.g., color from among a selection ofprimary and secondary colors, marker shape, etc.) to convey discreteinformation (e.g., alignment of an instrument trajectory and a referenceposition, intersection of an instrument and an image plane, etc.).Controller 311 may be configured to determine coded appearancecharacteristics of an indication, such as a graphical element, based onany combination of:

-   -   a location of the corresponding ultrasound operating environment        feature, or projection thereof, relative to the field-of-view of        the ultrasound image;    -   a location of the corresponding ultrasound operating environment        feature, or projection thereof, relative to the plane of the        ultrasound image;    -   a displacement between the corresponding ultrasound operating        environment feature, or projection thereof, and another feature        or projection;    -   alignment of the corresponding ultrasound operating environment        feature, or projection thereof, and another feature or        projection;    -   alignment of the corresponding ultrasound operating environment        feature and the plane of an ultrasound image (e.g., the plane of        an ultrasound image currently displayed on display 313);    -   an angle at which a corresponding ultrasound operating        environment feature intersects the plane of an ultrasound image        (e.g., an angle between the longitudinal axis of needle 317 and        the plane of an image acquired by probe 312);    -   a distance between a reference position and an image acquired by        probe 312 along a projector from the reference position to the        image that is orthogonal to the plane of the image;    -   a distance between a reference position and an image acquired by        probe 312 along a projector from the reference position to the        image that is parallel to a longitudinal axis of needle 317;    -   a distance between the location of an intersection of needle 317        with an arbitrary plane and an image acquired by probe 312 along        a projector from the intersection to the image that is        orthogonal to the plane of the image;    -   a distance between the location of an intersection of needle 317        with an arbitrary plane and an image acquired by probe 312 along        a projector from the intersection to the image that is parallel        to the longitudinal axis of the instrument; and    -   the like.        For example, in some embodiments, controller 311 is configured        to display a line representing the projection of the trajectory        of needle 317 onto the plane of the ultrasound image with a        first intensity when the intersection of the trajectory and the        plane of the ultrasound image lies outside the field-of-view of        the ultrasound image and with a second intensity when the        intersection lies within the field-of-view of the ultrasound        image.

In some embodiments, position base unit 305 is configured to monitor thequality of the tracking signal(s) from marker(s) on probe 312 and/orneedle 317. Position base unit 305 may provide an indication of thequality of such tracking signal(s) to controller 311. Position base unit305 may infer or estimate the quality of the tracking signal(s) from,for example, the mean received power of the tracking signal(s), theerror rate in a known training sequence, or the like.

In some embodiments, controller 311 is configured to display a graphicalor numerical indicator of the quality of tracking signals on display313. Graphical indicators of tracking signal quality may comprise, forexample, bar graphs, pivoting needles, or the like.

In some embodiments, controller 311 is configured to generate an alertwhen tracking signal quality falls below a threshold. A tracking signalquality threshold may comprise, for example, a mean received powerlevel, an error rate, or the like. An alert generated by controller 311may comprise, for example,

-   -   a textual warning message displayed on display 313;    -   an audible tone;    -   a verbal warning message;    -   a change in the appearance (e.g., attributes of color,        intensity, shape, linestyle, or the like) of some or all of the        display elements shown on display 313;    -   causing some or all of the display elements shown on display 313        to be hidden; or    -   the like.        In some embodiments, controller 311 is configured to cause lines        representing projections of needle 317 and its trajectory onto        the plane of an ultrasound image to disappear from display 313        when tracking signal quality is below a threshold.

FIG. 32 shows a portion 320 of an ultrasound system according to anexample embodiment. System 320 comprises a controller 321 connected to a3D position sensing system 325. Controller 321 comprises a positionsensing system interface 322, a display controller 323 and a positioninformation quality monitor 324. Position sensing system 325 comprises aposition signal receiver 326 configured to receive position signals frommarkers (not shown) rigidly connected to a needle 328 and an ultrasoundprobe 329. The position markers provide position signal receiver 326with tracking signals indicative of their location and/or orientation.Position sensing system 325 also comprises position base unit 327, whichis communicatively coupled to position signal receiver 326.

Position sensing system interface 322 is communicatively coupled toposition sensing system 325. Position sensing system interface 322provides information to controller 321 that is indicative of theposition and orientation of needle 327 and ultrasound probe 328, andindicative of the quality of the position and orientation information.

Display controller 323 is operable to generate data for driving adisplay (not shown). Display controller 323 is operable to generate datafor driving a display to show an image generated from ultrasound data(e.g., from ultrasound data acquired by probe 329) and to generate datafor driving a display to show graphics indicative of features of theenvironment. In FIG. 32, the details of how ultrasound data is providedto controller 321 and/or display controller 323 and the details of howcontroller 321 and/or display controller 323 interface with a display(not shown) are omitted for clarity. Display controller 323 comprises animage display inhibitor 323A, an image display appearance control 323B,a graphics display inhibitor 323C, a graphics display appearance controlunit 323D and a signal quality graphics generator 323E.

Position signal receiver 326 provides location and/or orientationinformation concerning the position markers to position base unit 327.Position signal receiver 326 may relay the tracking signals provided bythe position markers to position base unit 327. In some embodiments,position signal receiver 326 processes tracking signals from theposition markers to generate location and/or orientation informationconcerning the position markers, and provides this information toposition base unit 327.

In some embodiments, position signal receiver 326 is configured tomonitor the quality of the tracking signals from markers on probe 329and/or needle 328. Position signal receiver 326 may be configured toprovide an indication of the quality of the tracking signals to positionbase unit 327. Position signal receiver 326 may be configured to inferor estimate the quality of the tracking signals from, for example, themean received power of the tracking signals, an error rate in a knowntraining sequence, or the like. Position signal receiver 326 may beconfigured to communicate a tracking signal quality indicator, such as,for example, mean received power, error rate, a computed quality measureor the like, to position base unit 327. Position signal receiver 326 maybe configured to communicate tracking signal quality indicators that arespecific to particular markers or particular instruments (e.g., probe329 or needle 328).

In some embodiments, position signal receiver 326 is configured todetermine whether the quality of one or more of the tracking signalsfrom markers on probe 329 and/or needle 328 is below a threshold. Such athreshold may comprise, for example, a received power value, an errorrate, or the like. Position signal receiver 326 may be configured toprovide position base unit 327 with a low signal quality indication whenit determines that the quality of one or more of the tracking signalsfrom markers on probe 329 and/or needle 328 is below a threshold. A lowsignal quality indication may be specific to a particular marker whosetracking signal quality is below a threshold or be specific to aparticular instrument (e.g., probe 329 or needle 328) for which a markermounted thereon has a tracking signal quality below a threshold. A lowsignal quality indication may comprise position signal receiver 326 notproviding location and/or orientation information for one or moreposition markers to position base unit 327.

In some embodiments, position base unit 327 is configured to monitor thequality of the tracking signals from markers on probe 329 and/or needle328. Position base unit 327 may be configured to provide an indicationof the quality of the tracking signals to position sensing systeminterface 322. Position base unit 327 may be configured to infer orestimate the quality of the tracking signals from, for example, the meanreceived power of the tracking signals, the error rate in a knowntraining sequence, a tracking signal quality indicator provided byposition signal receiver 326, or the like. Position base unit 327 may beconfigured to communicate a tracking signal quality indicator, such as,for example, mean received power, error rate, a computed qualitymeasure, a tracking signal quality indicator generated by positionsignal receiver 326, or the like, to position sensing system interface322. Position base unit 327 may be configured to communicate trackingsignal quality indicators that are specific to particular markers orparticular instruments (e.g., probe 329 or needle 328).

In some embodiments, position base unit 327 is configured to determinewhether the quality of one or more of the tracking signals from markerson probe 329 and/or needle 328 is below a threshold. Such a thresholdmay comprise, for example, a received power value, an error rate, or thelike. Position base unit 327 may be configured to provide positionsensing system interface 322 with a low signal quality indication whenit determines that the quality of one or more of the tracking signalsfrom markers on probe 329 and/or needle 328 is below a threshold. A lowsignal quality indication may be specific to a particular marker whosetracking signal quality is below a threshold or be specific to aparticular instrument (e.g., probe 329 or needle 328) for which a markermounted thereon has a tracking signal quality below a threshold. A lowsignal quality indication may comprise position base unit 327 notproviding location and/or orientation information for one or moreposition markers to position sensing system interface 322.

Position sensing system interface 322 is configured to receive signalsfrom position base unit 327. Position sensing system interface 322 maybe configured to receive a tracking signal quality indicator and/or alow signal quality indication from position base unit 327. In someembodiments, position information quality monitor 324 is configured todetermine whether the quality of one or more of the tracking signalsfrom markers on probe 329 and/or needle 328 is below a threshold. Such athreshold may comprise, for example, a received power value, an errorrate, a computed quality measure value, or the like. For example,position information quality monitor 324 may be configured to determinewhether a tracking signal quality indicator provided by position basedunit 327 is below a threshold.

Position information quality monitor 324 of controller 321 is configuredto cause controller 321 to generate an alert when tracking signalquality is below a threshold, such as when a tracking signal qualityindication is below a threshold, when a low quality indication isreceived from position base unit 327, or in like circumstances.Controller 321 may be configured to generate visual and/or audiblealerts.

In some embodiments, controller 321 is configured to generate a visualalert by causing display controller 323 to change an aspect of thedisplay driving data it generates. Display controller 323 may beconfigured to change an aspect of the display driving data it generatesby doing one or more of the following:

-   -   causing image display inhibitor 323A to inhibit display of the        ultrasound image;    -   causing image display appearance controller 323B to change an        appearance of the ultrasound image;    -   causing graphics display inhibitor 323C to inhibit display of        one or more previously displayed graphics elements;    -   causing graphics display appearance controller 323D to change an        appearance (e.g., attributes of color, intensity, shape,        linestyle, or the like) of one or more previously displayed        graphics elements;    -   causing signal quality graphics generator 323E to generate a        graphic element indicative of the low tracking signal quality        condition, such as, for example, a textual message, or the like;        or    -   the like.

FIG. 33 shows an ultrasound system 370 according to an exampleembodiment. Ultrasound system 370 is useful for locating a referenceposition located in the body of patient P. An ultrasound probe 372 isoperable to receive ultrasound echo signals returning from a portion ofthe body of a patient P. An ultrasound image processor 374 iscommunicatively coupled to ultrasound probe 372. Ultrasound imageprocessor 374 is operable to generate an ultrasound image based on theultrasound echo signals received by ultrasound probe 372. A positionsensing system 376 is operable to determine a spatial location andorientation of ultrasound probe 372. An image plane locator 378 iscommunicatively coupled to position sensing system 376. Image planelocator 378 is operable to determine a spatial description of a plane ofthe ultrasound image generated by ultrasound image processor 374 fromthe ultrasound echo signals received by ultrasound probe 372 based onthe spatial location and orientation of the ultrasound probe 372determined by ultrasound position sensing system 376. A geometrycomputer 380 is communicatively coupled to image plane locator 378 and amemory 382. Memory 382 is operable to contain a spatial description ofthe reference position. Geometry computer 380 is operable to determine aspatial relationship between the reference position and the plane of theultrasound image based on the spatial description of the referenceposition in memory 382 and the spatial description of the plane of theultrasound image determined by image plane locator 378. A graphicsprocessor 384 is communicatively coupled to geometry computer 380.Graphics processor 384 is operable to generate a marker indicative ofthe spatial relationship, determined by geometry computer 380, betweenthe reference position and the plane of the ultrasound image. A display386 is communicatively coupled to ultrasound image processor 374 andgraphics processor 384. Display 386 is operable to display theultrasound image generated by ultrasound image processor 374 and themarker generated by graphics processor 384.

It is not mandatory that the ultrasound scans result in two-dimensionalimages (as are commonly obtained, for example, in B-mode imaging). Insome embodiments, the ultrasound scans obtain three-dimensional(volumetric) images.

The techniques described herein are not limited to B-mode ultrasoundscans.

Ultrasound imaging may be performed in other modes. For example, theultrasound imaging may be performed in modes such as:

-   -   Doppler modes;    -   color Doppler modes;    -   elastography;    -   etc.

In some embodiments, the 3D model is made up of ultrasound imagesacquired in a plurality of different modes. Or, equivalently, multiple3D models sharing the same coordinate system or having coordinatesystems that can be related to one another by some known transformationare provided for different ultrasound imaging modes. In suchembodiments, the apparatus may be configured to provide a 3D image thatcombines information from two or more different modes. A user controlmay be provided, for example, to control the blending of the informationin the displayed image. This may permit a user to identify specificanatomical features that are more readily visible to some modes ofultrasound imaging than they are to others.

It can be appreciated that the apparatus and methods described hereinhave application in a wide range of ultrasound imaging applications. Forexample, the methods and apparatus may be applied to:

-   -   obtaining biopsy samples;    -   placing radioactive seeds for cancer treatment or the like;    -   placing electrodes;    -   injecting drugs at specific locations;    -   inserting an epidural catheter, for example for the introduction        of an anaesthetic;    -   injecting epidural anaesthetic;    -   positioning surgical tools for minimally-invasive surgery;    -   etc.

Certain implementations of the invention comprise computer processorswhich execute software instructions which cause the processors toperform a method of the invention. For example, one or more processorsin an ultrasound system may implement methods as described above byexecuting software instructions in a program memory accessible to theprocessor(s). The invention may also be provided in the form of aprogram product. The program product may comprise any medium whichcarries a set of computer-readable signals comprising instructionswhich, when executed by a data processor, cause the data processor toexecute a method of the invention. Program products according to theinvention may be in any of a wide variety of forms. The program productmay comprise, for example, physical media such as magnetic data storagemedia including floppy diskettes, hard disk drives, optical data storagemedia including CD ROMs, DVDs, electronic data storage media includingROMs, flash RAM, or the like. The computer-readable signals on theprogram product may optionally be compressed or encrypted.

In addition to or as an alternative to performing methods by way ofsoftware executed in a programmable processor, such methods may beimplemented in whole or in part by suitable logic circuits. The logiccircuits may be provided in hard-wired form such as by hard-wired logiccircuits or one or more application specific integrated circuits. Thelogic circuits may in whole or part be provided by configurable logicsuch as suitably-configured field-programmable gate arrays.

Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to above, unless otherwise indicated,reference to that component (including a reference to a “means”) shouldbe interpreted as including as equivalents of that component anycomponent which performs the function of the described component (i.e.,that is functionally equivalent), including components which are notstructurally equivalent to the disclosed structure which performs thefunction in the illustrated exemplary embodiments of the invention.

Those skilled in the art will appreciate that certain features ofembodiments described herein may be used in combination with features ofother embodiments described herein, and that embodiments describedherein may be practised or implemented without all of the featuresascribed to them herein. Such variations on described embodiments thatwould be apparent to the skilled addressee, including variationscomprising mixing and matching of features from different embodiments,are within the scope of this invention.

As will be apparent to those skilled in the art in the light of theforegoing disclosure, many alterations, modifications, additions andpermutations are possible in the practice of this invention withoutdeparting from the spirit or scope thereof. The embodiments describedherein are only examples. Other example embodiments may be obtained,without limitation, by combining features of the disclosed embodiments.It is therefore intended that the following appended claims and claimshereafter introduced are interpreted to include all such alterations,modifications, permutations, additions, combinations andsub-combinations as are within their true spirit and scope.

What is claimed is:
 1. An ultrasound system for use in guiding medicalinterventions in a body, the ultrasound system comprising: an ultrasoundtransducer operable to receive ultrasound echo signals returning from aportion of the body; a fine elongate instrument adapted to be insertedin the body, the instrument defining a longitudinal axis, a positionsensing system operable to monitor a spatial location and orientation ofthe instrument and a spatial location and orientation of the ultrasoundtransducer; a controller communicatively coupled to the ultrasoundtransducer and the position sensing system; and a displaycommunicatively coupled to the controller; wherein the controller isconfigured to: generate a two dimensional ultrasound image based on theultrasound echo signals; display the ultrasound image on the display,the ultrasound image depicting a plane of the portion of the body;determine a location of the portion of the body depicted in theultrasound image based on the spatial location and orientation of theultrasound transducer; determine a first distance between a first linein the plane and a first point along the longitudinal axis at which thelongitudinal axis traverses the first line; determine a second distancebetween a second line in the plane and a second point along thelongitudinal axis at which the longitudinal axis traverses the secondline; generate on the display a first needle-image alignment indicatorhaving a first coded appearance characteristic indicative of the firstdistance; and generate on the display a second needle-image alignmentindicator having a second coded appearance characteristic indicative ofthe second distance.
 2. The ultrasound system of claim 1 wherein thecontroller is configured to: determine a first side of the plane whichthe first point along the longitudinal axis is on; determine a secondside of the plane which the second point along the longitudinal axis ison; wherein the first coded appearance characteristic is indicative ofthe first side and the second coded appearance characteristic isindicative of the second side.
 3. The ultrasound system of claim 2wherein the first line comprises a first edge of the portion of the bodydepicted in the ultrasound image and the second line comprises a secondedge of the portion of the body depicted in the ultrasound image.
 4. Theultrasound system of claim 2 wherein the first and second codedappearance characteristics each comprise a feature of shape.
 5. Theultrasound system of claim 2 wherein the first and second codedappearance characteristics are selected from a discrete set of differentcoded appearance characteristics and wherein the controller isconfigured to: determine the first coded appearance characteristic byselecting it from the discrete set of different coded appearancecharacteristics based at least in part on the first side; and determinethe second coded appearance characteristic by selecting it from thediscrete set of different coded appearance characteristics based atleast in part on the second side.
 6. The ultrasound system of claim 2wherein the first and second coded appearance characteristics of therespective first and second needle-image alignment indicators eachcomprise a combination of a color and a feature of shape.
 7. Theultrasound system of claim 6 wherein the controller is configured to:determine the color of the first needle-image alignment indicator basedat least in part on the first distance; determine the feature of shapeof the first needle-image alignment indicator based at least in part onthe first side; determine the color of the second needle-image alignmentindicator based at least in part on the second distance; and determinethe feature of shape of the second needle-image alignment indicatorbased at least in part on the second side.
 8. The ultrasound system ofclaim 1 wherein the controller is configured to: generate the firstneedle-image alignment indicator at a first location on the displayadjacent to a first line of the ultrasound image corresponding to thefirst line in the plane; and generate the second needle-image alignmentindicator at a second location on the display adjacent to a second lineof the ultrasound image corresponding to the second line in the plane.9. The ultrasound system claim 1 wherein the first and second codedappearance characteristics each comprise a color.
 10. The ultrasoundsystem of claim 1 wherein the first and second coded appearancecharacteristics each comprise a fill pattern.
 11. The ultrasound systemof claim 1 wherein the first and second coded appearance characteristicsare selected from a discrete set of different coded appearancecharacteristics.
 12. The ultrasound system of claim 11 wherein thecontroller is configured to: determine the first coded appearancecharacteristic by selecting it from the discrete set of different codedappearance characteristics based at least in part on whether the firstdistance is greater than a first threshold distance; and determine thesecond coded appearance characteristic by selecting it from the discreteset of different coded appearance characteristics based at least in parton whether the second distance is greater than a second thresholddistance.
 13. The ultrasound system of claim 12 wherein the controlleris configured to determine the first and second threshold distancesbased at least in part on a maximum angular separation between thelongitudinal axis and the plane.
 14. A method for providing a displayfor use in guiding a fine elongate instrument, the instrument defining alongitudinal axis, the method comprising: receiving ultrasound echosignals returning from a portion of the body; determining a spatiallocation and orientation of the instrument and a spatial location andorientation of the ultrasound transducer; generating a two dimensionalultrasound image based on the ultrasound echo signals; displaying theultrasound image on the display, the ultrasound image depicting a planeof the portion of the body; determining a location of the ultrasoundimage based on the spatial location and orientation of the ultrasoundtransducer; determining a location of the portion of the body depictedin the ultrasound image based on the spatial location and orientation ofthe ultrasound transducer; determining a first distance between a firstline in the plane and a first point along the longitudinal axis at whichthe longitudinal axis traverses the first line; determining a seconddistance between a second line in the plane and a second point along thelongitudinal axis at which the longitudinal axis traverses the secondline; generating on the display a first needle-image alignment indicatorhaving a first coded appearance characteristic indicative of the firstdistance; and generating on the display a second needle-image alignmentindicator having a second coded appearance characteristic indicative ofthe second distance.
 15. The method of claim 14 comprising: determininga first side of the plane which the first point along the longitudinalaxis is on; determining a second side of the plane which the secondpoint along the longitudinal axis is on; wherein the first codedappearance characteristic is indicative of the first side and the secondcoded appearance characteristic is indicative of the second side. 16.The method of claim 15 wherein the first and second coded appearancecharacteristics each comprise a feature of shape.
 17. The method ofclaim 15 comprising: determining the first coded appearancecharacteristic by selecting it from the discrete set of different codedappearance characteristics based on the first side; and determining thesecond coded appearance characteristic by selecting it from the discreteset of different coded appearance characteristics based on the secondside.
 18. The method of claim 15 wherein the first and second codedappearance characteristics of the respective first and secondneedle-image alignment indicators each comprise a combination of a colorand a feature of shape.
 19. The method of claim 18 comprising:determining the color of the first needle-image alignment indicatorbased at least in part on the first distance; determining the feature ofshape of the first needle-image alignment indicator based at least inpart on the first side; determining the color of the second needle-imagealignment indicator based at least in part on the second distance; anddetermining the feature of shape of the second needle-image alignmentindicator based at least in part on the second side.
 20. The method ofclaim 14 wherein the first line comprises a first edge of the portion ofthe body depicted in the ultrasound image and the second line comprisesa second edge of the portion of the body depicted in the ultrasoundimage.
 21. The method of claim 14 comprising: generating the firstneedle-image alignment indicator at a first location on the displayadjacent to a first line of the ultrasound image corresponding to thefirst line in the plane; and generating the second needle-imagealignment indicator at a second location on the display adjacent to asecond line of the ultrasound image corresponding to the second line inthe plane.
 22. The method of claim 14 wherein the first and second codedappearance characteristics each comprise a color.
 23. The method ofclaim 14 wherein the first and second coded appearance characteristicseach comprise a fill pattern.
 24. The method of claim 14 wherein thefirst and second coded appearance characteristics are selected from adiscrete set of different coded appearance characteristics.
 25. Themethod of claim 24 comprising: determining the first coded appearancecharacteristic by selecting it from the discrete set of different codedappearance characteristics based at least in part on whether the firstdistance is greater than a first threshold distance; and determining thesecond coded appearance characteristic by selecting it from the discreteset of different coded appearance characteristics based at least in parton whether the second distance is greater than a second thresholddistance.
 26. The method of claim 25 comprising determining the firstand second threshold distances based at least in part on a maximumangular separation between the longitudinal axis and the plane of theultrasound image.